CUMULUS-3104: Fixed TS compilation error caused by @aws-sdk/client-s3 upgrade #3132

jennyhliu · 2022-10-21T14:38:19Z

Summary: Summary of changes

Addresses CUMULUS-XX: Develop amazing new feature

Changes

Fixed TS compilation error caused by @aws-sdk/client-s3 3.190->3.193 upgrade

> @cumulus/ingest@13.3.2 tsc:listEmittedFiles
> ../../node_modules/.bin/tsc --listEmittedFiles

src/S3ProviderClient.ts:135:9 - error TS2322: Type '"private"' is not assignable to type 'ObjectCannedACL | undefined'.

135         ACL: 'private',
            ~~~

  ../aws-client/S3.d.ts:421:5
    421     ACL?: ObjectCannedACL;
            ~~~
    The expected type comes from property 'ACL' which is declared here on type '{ sourceBucket: string; sourceKey: string; destinationBucket: string; destinationKey: string; sourceObject?: HeadObjectOutput | undefined; ACL?: ObjectCannedACL | undefined; copyTags?: boolean | undefined; chunkSize?: number | undefined; }'


Found 1 error in src/S3ProviderClient.ts:135

Skip the unit tests which calls S3 deleteObjects. tests/test-S3.js and tests/lambdas/test-process-s3-dead-letter-archive.js, the updated @AWS-SDK doesn't work with localstack/localstack:0.12.13. Add ticket https://bugs.earthdata.nasa.gov/browse/CUMULUS-3106

TypeError: Cannot read property '#text' of undefined
    at /Users/jhliu/cumulus/packages/aws-client/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:10083:30
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at parseErrorBody (/Users/jhliu/cumulus/packages/aws-client/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:10093:19)
    at deserializeAws_restXmlDeleteObjectsCommandError (/Users/jhliu/cumulus/packages/aws-client/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3606:15)
    at /Users/jhliu/cumulus/packages/aws-client/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at /Users/jhliu/cumulus/packages/aws-client/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
    at StandardRetryStrategy.retry (/Users/jhliu/cumulus/packages/aws-client/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
    at /Users/jhliu/cumulus/packages/aws-client/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:56:20
    at /Users/jhliu/cumulus/packages/aws-client/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
    at /Users/jhliu/cumulus/packages/aws-client/tests/test-S3.js:109:5 {
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}
  ✖ deleteS3Objects() deletes s3 objects Rejected promise returned by test

PR Checklist

Update CHANGELOG
Unit tests
Ad-hoc testing - Deploy changes and test manually
Integration tests

…upgrade

jennyhliu · 2022-10-24T14:06:16Z

bamboo/cleanup-integration-tests.sh

@@ -20,6 +20,6 @@ npm install
 npm --version

 ## This is needed to ensure lock-stack has the expected dependencies
-npx lerna bootstrap --scope @cumulus/cumulus-integration-tests --scope @cumulus/aws-client --scope @cumulus/common --scope @cumulus/errors --scope @cumulus/logger
+npx lerna bootstrap --scope @cumulus/cumulus-integration-tests --scope @cumulus/aws-client --scope @cumulus/checksum --scope @cumulus/common --scope @cumulus/errors --scope @cumulus/logger


fix ci post build error.

jennyhliu · 2022-10-24T14:07:02Z

example/lambdas/lzardsClientTest/package.json

-    "@cumulus/lzards-api-client": "13.2.2-alpha.2",
-    "@cumulus/logger": "13.3.0"
+    "@cumulus/lzards-api-client": "13.3.2-alpha.0",
+    "@cumulus/logger": "13.3.2"


correct the package version

jennyhliu · 2022-10-24T14:07:55Z

package.json

@@ -68,7 +68,6 @@
  "author": "Cumulus Authors",
  "license": "Apache-2.0",
  "dependencies": {
-    "@cumulus/aws-client": "^1.24.0",


remove the unnecessary old package which could cause the compile error and audit error,
add missing packages for packages/async-operations/package.json, lambdas/postgres-migration-count-tool/package.json

"@cumulus/aws-client": "^1.24.0"
It causes issue when I try to build cumulus/aws-client, npm run tsc:listEmittedFiles
src/S3.ts:63:14 - error TS2339: Property 'buildMessage' does not exist on type 'Logger'.
63 return log.buildMessage('warn', message);

jennyhliu · 2022-10-24T14:12:52Z

packages/api/tests/lambdas/test-process-s3-dead-letter-archive.js

@@ -68,7 +68,7 @@ test.after(async (t) => {
  await S3.recursivelyDeleteS3Bucket(t.context.bucket);
 });

-test('processDeadLetterArchive calls writeRecords for each dead letter Cumulus message', async (t) => {
+test.skip('processDeadLetterArchive calls writeRecords for each dead letter Cumulus message', async (t) => {


skip unit tests which call @aws-sdk/client-s3 S3 deleteObjects. @aws-sdk/client-s3@3.193.0 doesn't work with localstack/localstack:0.12.13

Should we add a comment to explain this change?

botanical · 2022-10-24T20:30:23Z

packages/aws-client/tests/test-S3.js

@@ -116,6 +116,8 @@ test('deleteS3Objects() deletes s3 objects', async (t) => {
      Bucket: bucketName,
    });
    t.is(objects2.length, 0);
+  } catch (e) {
+    console.log(e);


Do we still need this change if we are skipping this test?

botanical

Had a couple comments but the main work done looks good

* initial commit * add postgres query to LIST endpoints * add tests for granules and collections * Release 13.3.x with master (#3121) * Release 13.3.1 (#3106) * merge CUMULUS-2557 * merge CUMULUS-2971 * CUMULUS-3021: Support Collections with a collection.version containing slashes (#3096) * Support Collections with a collection.version containing slashes * merge CUMULUS-3021 * merge CUMULUS-3024 * bump version to 13.3.1 * update documentation * update changelog * remove duplicate api user * Sort results in unit test before comparison Co-authored-by: Charles Huang <charleshuang80@users.noreply.github.com> Co-authored-by: jennyhliu <34660846+jennyhliu@users.noreply.github.com> Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> * CUMULUS-3027 -- Tightly constrain typescript version due to knex typing (#3095) * Tightly constrain typescript verseion due to knex typing Unpinned dependency resulted in typescript 4.8 being installed, however knex (all versions) has an outstanding bug: knex/knex#5279 * Add CHANGELOG 🔔 * Update generate-ts-build-cache to always 'npm install' * Update CHANGELOG * Release 13.3.2 (#3117) * Update CNM lambdas to utilize newer releases (#3094) * 13.3.2 version bump * update changelog Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> * update changelog * re-add comma Co-authored-by: Charles Huang <charleshuang80@users.noreply.github.com> Co-authored-by: jennyhliu <34660846+jennyhliu@users.noreply.github.com> Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> * fix lint error * CUMULUS-3104:Update async operation image Dockerfile (#3130) * CUMULUS-3104:Update async operation image Dockerfile to use node:14.19.3-buster Upgraded saml2-js 4.0.0, rewire to 6.0.0 to address security vulnerabilities * add xmldom to audit allowlist * update default async_operation_image * update changelog skip-integration-tests * CUMULUS-2944: granules/bulkDelete endpoint has PayloadTooLargeError for 4k granules (#3122) * Update body parser configuration to increase limit * Add unit test, update changelog * Update unit test and fix lint * Fix lint errors * Fix lint error * Relocate granulesList.js and update reference in unit test * Update helper function to dynamically generate IDs and move to different file. Update unit test * Fix lint errors * Increase limit to match lambda limit * CUMULUS-3102: Update broken link (#3125) * Fix docs that has bad link * Update url to reference markdown file directly * CUMULUS-3104: Fixed TS compilation error caused by @aws-sdk/client-s3 upgrade (#3132) * Fixed TS compilation error caused by @aws-sdk/client-s3 3.190->3.193 upgrade * fix cumulus/package.json * update lerna to fix security vulnerability * fix lzards-api-client package * temporarily skip deleteS3Objects test * remove @Cumulus from root, skip test deleteObjects, add missing package * fix ci post build failure * put xmldom back to allow list * add comments for skipped test Co-authored-by: Charles Huang <charleshuang80@users.noreply.github.com> Co-authored-by: jennyhliu <34660846+jennyhliu@users.noreply.github.com> Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> Co-authored-by: Jennifer Tran <12633533+botanical@users.noreply.github.com>

… upgrade (#3132) * Fixed TS compilation error caused by @aws-sdk/client-s3 3.190->3.193 upgrade * fix cumulus/package.json * update lerna to fix security vulnerability * fix lzards-api-client package * temporarily skip deleteS3Objects test * remove @Cumulus from root, skip test deleteObjects, add missing package * fix ci post build failure * put xmldom back to allow list * add comments for skipped test

* Updated Changelog * 11.1.8 Version bump * Fix vulnerability * CUMULUS-3104: Fixed TS compilation error caused by @aws-sdk/client-s3 upgrade (#3132) * Fixed TS compilation error caused by @aws-sdk/client-s3 3.190->3.193 upgrade * fix cumulus/package.json * update lerna to fix security vulnerability * fix lzards-api-client package * temporarily skip deleteS3Objects test * remove @Cumulus from root, skip test deleteObjects, add missing package * fix ci post build failure * put xmldom back to allow list * add comments for skipped test * CUMULUS-3104:Update async operation image Dockerfile (#3130) * CUMULUS-3104:Update async operation image Dockerfile to use node:14.19.3-buster Upgraded saml2-js 4.0.0, rewire to 6.0.0 to address security vulnerabilities * add xmldom to audit allowlist * update default async_operation_image * update changelog skip-integration-tests * CUMULUS-3104-2: Fixed TS compilation error caused by @aws-sdk/client-s3 3.202.0 upgrade (#3142) * CUMULUS-3104:Fix tsc:listEmittedFiles SQS type error on aws-client * update change log * use ?? instead of || * add debug error * Resolving conflicts * Resolving conflicts * Fixing lint errors * Adding deployment stack * Adding Dependency * [CUMULUS-2903]: Bump Node version to 14.19.1 (#2917) (#3152) * bump node version to 14.19.1 AWS Lambda supports 14.x as the most recent Node runtime. 14.19.1 is the latest minor version release for Node 14. * Update docs for Node version bump * Revert doc update * update changelog * Update changelog with esc-task instruction * Add ES execution cleanup to prevent orphaned ES records after test runs * Update lambda.md * update ecs task version Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> Co-authored-by: Nate Pauzenga <npauzenga@gmail.com> Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> Co-authored-by: Naga Nages <66387215+Nnaga1@users.noreply.github.com> Co-authored-by: Katherine Kelly <katherine.a.kelly@nasa.gov> Co-authored-by: jennyhliu <34660846+jennyhliu@users.noreply.github.com> Co-authored-by: Nate Pauzenga <npauzenga@gmail.com> Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org>

… upgrade (#3132) * Fixed TS compilation error caused by @aws-sdk/client-s3 3.190->3.193 upgrade * fix cumulus/package.json * update lerna to fix security vulnerability * fix lzards-api-client package * temporarily skip deleteS3Objects test * remove @Cumulus from root, skip test deleteObjects, add missing package * fix ci post build failure * put xmldom back to allow list * add comments for skipped test

* merge CUMULUS-2557 * merge CUMULUS-2971 * merge CUMULUS-3021 * merge CUMULUS-3024 * 11.1.6 version bump * Release 11.1.6 (#3109) * CUMULUS-3027 * Update CHANGELOG.md Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> * Update CNM lambdas to utilize newer releases (#3094) * Sort results in unit test before comparison * update version to 11.1.7 * Release 11.1.8 node 14 (#3144) * Updated Changelog * 11.1.8 Version bump * Fix vulnerability * CUMULUS-3104: Fixed TS compilation error caused by @aws-sdk/client-s3 upgrade (#3132) * Fixed TS compilation error caused by @aws-sdk/client-s3 3.190->3.193 upgrade * fix cumulus/package.json * update lerna to fix security vulnerability * fix lzards-api-client package * temporarily skip deleteS3Objects test * remove @Cumulus from root, skip test deleteObjects, add missing package * fix ci post build failure * put xmldom back to allow list * add comments for skipped test * CUMULUS-3104:Update async operation image Dockerfile (#3130) * CUMULUS-3104:Update async operation image Dockerfile to use node:14.19.3-buster Upgraded saml2-js 4.0.0, rewire to 6.0.0 to address security vulnerabilities * add xmldom to audit allowlist * update default async_operation_image * update changelog skip-integration-tests * CUMULUS-3104-2: Fixed TS compilation error caused by @aws-sdk/client-s3 3.202.0 upgrade (#3142) * CUMULUS-3104:Fix tsc:listEmittedFiles SQS type error on aws-client * update change log * use ?? instead of || * add debug error * Resolving conflicts * Resolving conflicts * Fixing lint errors * Adding deployment stack * Adding Dependency * [CUMULUS-2903]: Bump Node version to 14.19.1 (#2917) (#3152) * bump node version to 14.19.1 AWS Lambda supports 14.x as the most recent Node runtime. 14.19.1 is the latest minor version release for Node 14. * Update docs for Node version bump * Revert doc update * update changelog * Update changelog with esc-task instruction * Add ES execution cleanup to prevent orphaned ES records after test runs * Update lambda.md * update ecs task version Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> Co-authored-by: Nate Pauzenga <npauzenga@gmail.com> Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> Co-authored-by: Naga Nages <66387215+Nnaga1@users.noreply.github.com> Co-authored-by: Katherine Kelly <katherine.a.kelly@nasa.gov> Co-authored-by: jennyhliu <34660846+jennyhliu@users.noreply.github.com> Co-authored-by: Nate Pauzenga <npauzenga@gmail.com> Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> * Update Changelog (#3155) Co-authored-by: Katherine Kelly <katherine.a.kelly@nasa.gov> * Fixing CHANGELOG Co-authored-by: Charles Huang <charleshuang80@users.noreply.github.com> Co-authored-by: jennyhliu <34660846+jennyhliu@users.noreply.github.com> Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> Co-authored-by: Nate Pauzenga <npauzenga@gmail.com> Co-authored-by: jennyhliu <jenny.h.liu@nasa.gov> Co-authored-by: Naga Nages <66387215+Nnaga1@users.noreply.github.com> Co-authored-by: Katherine Kelly <katherine.a.kelly@nasa.gov>

jennyhliu added 2 commits October 21, 2022 10:37

Fixed TS compilation error caused by @aws-sdk/client-s3 3.190->3.193 …

7521e74

…upgrade

fix cumulus/package.json

6e2aa6f

jennyhliu force-pushed the jl/fix-acl-ts-error branch from 120e624 to 6e2aa6f Compare October 21, 2022 16:02

jennyhliu added 6 commits October 21, 2022 12:27

update lerna to fix security vulnerability

69dcf98

fix lzards-api-client package

4f0a085

temporarily skip deleteS3Objects test

d74be5d

remove @Cumulus from root, skip test deleteObjects, add missing package

97744db

fix ci post build failure

ac25772

put xmldom back to allow list

1b25b86

jennyhliu commented Oct 24, 2022

View reviewed changes

jennyhliu added needs review high priority labels Oct 24, 2022

botanical added in review and removed needs review labels Oct 24, 2022

botanical reviewed Oct 24, 2022

View reviewed changes

botanical added pr feedback and removed in review labels Oct 24, 2022

botanical approved these changes Oct 24, 2022

View reviewed changes

add comments for skipped test

9ccd659

jennyhliu merged commit 56e2374 into master Oct 24, 2022

jennyhliu deleted the jl/fix-acl-ts-error branch October 24, 2022 23:09

jennyhliu removed pr feedback high priority labels Oct 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUMULUS-3104: Fixed TS compilation error caused by @aws-sdk/client-s3 upgrade #3132

CUMULUS-3104: Fixed TS compilation error caused by @aws-sdk/client-s3 upgrade #3132

jennyhliu commented Oct 21, 2022 •

edited

jennyhliu Oct 24, 2022

jennyhliu Oct 24, 2022

jennyhliu Oct 24, 2022 •

edited

jennyhliu Oct 24, 2022

botanical Oct 24, 2022

botanical Oct 24, 2022

botanical left a comment

CUMULUS-3104: Fixed TS compilation error caused by @aws-sdk/client-s3 upgrade #3132

CUMULUS-3104: Fixed TS compilation error caused by @aws-sdk/client-s3 upgrade #3132

Conversation

jennyhliu commented Oct 21, 2022 • edited

Changes

PR Checklist

jennyhliu Oct 24, 2022

Choose a reason for hiding this comment

jennyhliu Oct 24, 2022

Choose a reason for hiding this comment

jennyhliu Oct 24, 2022 • edited

Choose a reason for hiding this comment

jennyhliu Oct 24, 2022

Choose a reason for hiding this comment

botanical Oct 24, 2022

Choose a reason for hiding this comment

botanical Oct 24, 2022

Choose a reason for hiding this comment

botanical left a comment

Choose a reason for hiding this comment

jennyhliu commented Oct 21, 2022 •

edited

jennyhliu Oct 24, 2022 •

edited