Skip to content

v1.18.0
Compare
Choose a tag to compare
@laurenfrederick laurenfrederick released this 03 Feb 22:31
· 3762 commits to master since this release
v1.18.0
fe4edaf

This is a fairly minor breaking release. See migration steps below for required changes.

Migration Steps

  • If not already set, set ecs_cluster_instance_image_id in your terraform.tfvars for your cumulus module deployment

  • Change variable saml_launchpad_metadata_path to saml_launchpad_metadata_url in the cumulus Terraform module.

  • Remove process from PostToCmr task_config in workflow definitions.

  • Please note function removals and deprecations listed below in the Removed and Deprecated sections.

Breaking Changes

  • CUMULUS-1686

    • ecs_cluster_instance_image_id is now a required variable of the cumulus module, instead of optional.

  • CUMULUS-1698

    • Change variable saml_launchpad_metadata_path to saml_launchpad_metadata_url in the tf-modules/cumulus Terraform module.

  • CUMULUS-1703

    • Remove the unused forceDownload option from the sync-granule tasks's config
    • Remove the @cumulus/ingest/granule.Discover class
    • Remove the @cumulus/ingest/granule.Granule class
    • Remove the @cumulus/ingest/pdr.Discover class
    • Remove the @cumulus/ingest/pdr.Granule class
    • Remove the @cumulus/ingest/parse-pdr.parsePdr function

Other Notable Changes

  • Upgrade Thin Egress App (TEA) to version 48
  • Fixed granule discovery recursion algorithm used in S/FTP protocols.

All Changes

Added

  • CUMULUS-1040

    • Added @cumulus/aws-client package to provide utilities for working with AWS services and the Node.js AWS SDK
    • Added @cumulus/errors package which exports error classes for use in Cumulus workflow code
    • Added @cumulus/integration-tests/sfnStep to provide utilities for parsing step function execution histories

  • CUMULUS-1102

    • Adds functionality to the @cumulus/api package for better local testing.
      • Adds data seeding for @cumulus/api's localAPI.
        • seed functions allow adding collections, executions, granules, pdrs, providers, and rules to a Localstack Elasticsearch and DynamoDB via addCollections, addExecutions, addGranules, addPdrs, addProviders, and addRules.
      • Adds eraseDataStack function to local API server code allowing resetting of local datastack for testing (ES and DynamoDB).
      • Adds optional parameters to the @cumulus/api bin serve to allow for launching the api without destroying the current data.

  • CUMULUS-1697

    • Added the @cumulus/tf-inventory package that provides command line utilities for managing Terraform resources in your AWS account

  • CUMULUS-1703

    • Add @cumulus/aws-client/S3.createBucket function
    • Add @cumulus/aws-client/S3.putFile function
    • Add @cumulus/common/string.isNonEmptyString function
    • Add @cumulus/ingest/FtpProviderClient class
    • Add @cumulus/ingest/HttpProviderClient class
    • Add @cumulus/ingest/S3ProviderClient class
    • Add @cumulus/ingest/SftpProviderClient class
    • Add @cumulus/ingest/providerClientUtils.buildProviderClient function
    • Add @cumulus/ingest/providerClientUtils.fetchTextFile function

  • CUMULUS-1731

    • Add new optional input variables to the Cumulus Terraform module to support TEA upgrade:
      • thin_egress_cookie_domain - Valid domain for Thin Egress App cookie
      • thin_egress_domain_cert_arn - Certificate Manager SSL Cert ARN for Thin
        Egress App if deployed outside NGAP/CloudFront
      • thin_egress_download_role_in_region_arn - ARN for reading of Thin Egress
        App data buckets for in-region requests
      • thin_egress_jwt_algo - Algorithm with which to encode the Thin Egress
        App JWT cookie
      • thin_egress_jwt_secret_name - Name of AWS secret where keys for the Thin
        Egress App JWT encode/decode are stored
      • thin_egress_lambda_code_dependency_archive_key - Thin Egress App - S3
        Key of packaged python modules for lambda dependency layer

  • CUMULUS-1733

    • Add discovery-filtering operator doc to document previously undocumented functionality.

Changed

  • CUMULUS-1102

    • Updates @cumulus/api/auth/testAuth to use JWT instead of random tokens.
    • Updates the default AMI for the ecs_cluster_instance_image_id.

  • CUMULUS-1622

    • Mutex class has been deprecated in @cumulus/common/concurrency and will be removed in a future release.

  • CUMULUS-1686

    • Changed ecs_cluster_instance_image_id to be a required variable of the cumulus module and removed the default value.
      The default was not available across accounts and regions, nor outside of NGAP and therefore not particularly useful.

  • CUMULUS-1688

    • Updated @cumulus/aws.receiveSQSMessages not to replace message.Body with a parsed object. This behavior was undocumented and confusing as received messages appeared to contradict AWS docs that state message.Body is always a string.
    • Replaced sf_watcher CloudWatch rule from cloudwatch-events.tf with an EventSourceMapping on sqs2sf mapped to the start_sf SQS queue (in event-sources.tf).
    • Updated sqs2sf with an EventSourceMapping handler and unit test.

  • CUMULUS-1698

    • Change variable saml_launchpad_metadata_path to saml_launchpad_metadata_url in the tf-modules/cumulus Terraform module.
    • Updated @cumulus/api/launchpadSaml to download launchpad IDP metadata from configured location when the metadata in s3 is not valid, and to work with updated IDP metadata and SAML response.

  • CUMULUS-1731

    • Upgrade the version of the Thin Egress App deployed by Cumulus to v48
      • Note: New variables available, see the 'Added' section of this changelog.

Fixed

  • CUMULUS-1664

    • Updated dbIndexer Lambda to remove hardcoded references to DynamoDB table names.

  • CUMULUS-1733

    • Fixed granule discovery recursion algorithm used in S/FTP protocols.

Removed

  • CUMULUS-1481

    • removed process config and output from PostToCmr as it was not required by the task nor downstream steps, and should still be in the output message's meta regardless.

  • CUMULUS-1703

    • Remove the unused forceDownload option from the sync-granule tasks's config
    • Remove the @cumulus/ingest/granule.Discover class
    • Remove the @cumulus/ingest/granule.Granule class
    • Remove the @cumulus/ingest/pdr.Discover class
    • Remove the @cumulus/ingest/pdr.Granule class
    • Remove the @cumulus/ingest/parse-pdr.parsePdr function

Deprecated

  • CUMULUS-1040
    • Deprecated the following code. For cases where the code was moved into another package, the new code location is noted:
      • @cumulus/common/CloudFormationGateway -> @cumulus/aws-client/CloudFormationGateway
      • @cumulus/common/DynamoDb -> @cumulus/aws-client/DynamoDb
      • @cumulus/common/errors -> @cumulus/errors
      • @cumulus/common/StepFunctions -> @cumulus/aws-client/StepFunctions
      • All of the exported functions in @cumulus/commmon/aws (moved into @cumulus/aws-client), except:
        • @cumulus/common/aws/isThrottlingException -> @cumulus/errors/isThrottlingException
        • @cumulus/common/aws/improveStackTrace (not deprecated)
        • @cumulus/common/aws/retryOnThrottlingException (not deprecated)
      • @cumulus/common/sfnStep/SfnStep.parseStepMessage -> @cumulus/integration-tests/sfnStep/SfnStep.parseStepMessage
      • @cumulus/common/sfnStep/ActivityStep -> @cumulus/integration-tests/sfnStep/ActivityStep
      • @cumulus/common/sfnStep/LambdaStep -> @cumulus/integration-tests/sfnStep/LambdaStep
      • @cumulus/common/string/unicodeEscape -> @cumulus/aws-client/StepFunctions.unicodeEscape
      • @cumulus/common/util/setErrorStack -> @cumulus/aws-client/util/setErrorStack
      • @cumulus/ingest/aws/invoke -> @cumulus/aws-client/Lambda/invoke
      • @cumulus/ingest/aws/CloudWatch.bucketSize
      • @cumulus/ingest/aws/CloudWatch.cw
      • @cumulus/ingest/aws/ECS.ecs
      • @cumulus/ingest/aws/ECS
      • @cumulus/ingest/aws/Events.putEvent -> @cumulus/aws-client/CloudwatchEvents.putEvent
      • @cumulus/ingest/aws/Events.deleteEvent -> @cumulus/aws-client/CloudwatchEvents.deleteEvent
      • @cumulus/ingest/aws/Events.deleteTarget -> @cumulus/aws-client/CloudwatchEvents.deleteTarget
      • @cumulus/ingest/aws/Events.putTarget -> @cumulus/aws-client/CloudwatchEvents.putTarget
      • @cumulus/ingest/aws/SQS.attributes -> @cumulus/aws-client/SQS.getQueueAttributes
      • @cumulus/ingest/aws/SQS.deleteMessage -> @cumulus/aws-client/SQS.deleteSQSMessage
      • @cumulus/ingest/aws/SQS.deleteQueue -> @cumulus/aws-client/SQS.deleteQueue
      • @cumulus/ingest/aws/SQS.getUrl -> @cumulus/aws-client/SQS.getQueueUrlByName
      • @cumulus/ingest/aws/SQS.receiveMessage -> @cumulus/aws-client/SQS.receiveSQSMessages
      • @cumulus/ingest/aws/SQS.sendMessage -> @cumulus/aws-client/SQS.sendSQSMessage
      • @cumulus/ingest/aws/StepFunction.getExecutionStatus -> @cumulus/aws-client/StepFunction.getExecutionStatus
      • @cumulus/ingest/aws/StepFunction.getExecutionUrl -> @cumulus/aws-client/StepFunction.getExecutionUrl
Assets 7
Loading