Skip to content

v1.11.2
Compare
Choose a tag to compare
@laurenfrederick laurenfrederick released this 18 Feb 21:10
· 6473 commits to master since this release
v1.11.2
d5bdf36

Notable Changes

Schemas are strictly enforced when writing to the database. Additional properties are not allowed.

UMM-G JSON Support:

  • Move granules task, post to CMR, and move granules API have been updated to handle all current versions of UMM-G
  • Note that if your collection definition points to URL paths using the Echo10 notation, it will need to be updated to UMM notation to put your files in the correct location

Backend API Refactor:

  • The backend API has been refactored to all be contained in one lambda
  • Express is the new API engine
  • See CUMULUS-1043 notes below for more information

Added

  • CUMULUS-1169
    • Added a @cumulus/common/StepFunctions module. It contains functions for querying the AWS
      StepFunctions API. These functions have the ability to retry when a ThrottlingException occurs.
    • Added @cumulus/common/aws.retryOnThrottlingException(), which will wrap a function in code to
      retry on ThrottlingExceptions.
    • Added @cumulus/common/test-utils.throttleOnce(), which will cause a function to return a
      ThrottlingException the first time it is called, then return its normal result after that.
  • CUMULUS-1103 Compare the collection holdings in CMR with Cumulus' internal data store
  • CUMULUS-1099 Add support for UMMG JSON metadata versions > 1.4.
    • If a version is found in the metadata object, that version is used for processing and publishing to CMR otherwise, version 1.4 is assumed.
  • CUMULUS-678
    • Added support for UMMG json v1.4 metadata files.
      reconcileCMRMetadata added to @cumulus/cmrjs to update metadata record with new file locations.
      @cumulus/common/errors adds two new error types CMRMetaFileNotFound and InvalidArgument.
      @cumulus/common/test-utils adds new function randomId to create a random string with id to help in debugging.
      @cumulus/common/BucketsConfig adds a new helper class BucketsConfig for working with bucket stack configuration and bucket names.
      @cumulus/common/aws adds new function s3PutObjectTagging as a convenience for the aws s3().putObjectTagging function.
      @cumulus/cmrjs Adds:
      • isCMRFile - Identify an echo10(xml) or UMMG(json) metadata file.
      • metadataObjectFromCMRFile Read and parse CMR XML file from s3.
      • updateCMRMetadata Modify a cmr metadata (xml/json) file with updated information.
      • publish2CMR Posts XML or UMMG CMR data to CMR service.
      • reconcileCMRMetadata Reconciles cmr metadata file after a file moves.
  • Adds some ECS and other permissions to StepRole to enable running ECS tasks from a workflow
  • Added Apache logs to cumulus api and distribution lambdas
  • CUMULUS-1119 - Added @cumulus/integration-tests/api/EarthdataLogin.getEarthdataLoginRedirectResponse helper for integration tests to handle login with Earthdata and to return response from redirect to Cumulus API
  • CUMULUS-673 Added @cumulus/common/file/getFileChecksumFromStream to get file checksum from a readable stream

Fixed

  • CUMULUS-1123
    • Cloudformation template overrides now work as expected

Changed

  • CUMULUS-1169

    • Deprecated the @cumulus/common/step-functions module.
    • Updated code that queries the StepFunctions API to use the retry-enabled functions from
      @cumulus/common/StepFunctions

  • CUMULUS-1121

    • Schema validation is now strongly enforced when writing to the database.
      Additional properties are not allowed and will result in a validation error.

  • CUMULUS-678
    tasks/move-granules simplified and refactored to use functionality from cmrjs.
    ingest/granules.moveGranuleFiles now just moves granule files and returns a list of the updated files. Updating metadata now handled by @cumulus/cmrjs/reconcileCMRMetadata.
    move-granules.updateGranuleMetadata refactored and bugs fixed in the case of a file matching multiple collection.files.regexps.
    getCmrXmlFiles simplified and now only returns an object with the cmrfilename and the granuleId.
    @cumulus/test-processing - test processing task updated to generate UMM-G metadata

  • CUMULUS-1043

    • @cumulus/api now uses express as the API engine.
    • All @cumulus/api endpoints on ApiGateway are consolidated to a single endpoint the uses {proxy+} definition.
    • All files under packages/api/endpoints along with associated tests are updated to support express's request and response objects.
    • Replaced environment variables internal, bucket and systemBucket with system_bucket.
    • Update @cumulus/integration-tests to work with updated cumulus-api express endpoints

  • @cumulus/integration-tests - buildAndExecuteWorkflow and buildWorkflow updated to take a meta param to allow for additional fields to be added to the workflow meta

  • CUMULUS-1049 Updated Retrieve Execution Status API in @cumulus/api: If the execution doesn't exist in Step Function API, Cumulus API returns the execution status information from the database.

  • CUMULUS-1119

    • Renamed DISTRIBUTION_URL environment variable to DISTRIBUTION_ENDPOINT
    • Renamed DEPLOYMENT_ENDPOINT environment variable to DISTRIBUTION_REDIRECT_ENDPOINT
    • Renamed API_ENDPOINT environment variable to TOKEN_REDIRECT_ENDPOINT

Removed

  • Functions deprecated before 1.11.0:
    • @cumulus/api/models/base: static Manager.createTable() and static Manager.deleteTable()
    • @cumulus/ingest/aws/S3
    • @cumulus/ingest/aws/StepFunction.getExecution()
    • @cumulus/ingest/aws/StepFunction.pullEvent()
    • @cumulus/ingest/consumer.Consume
    • @cumulus/ingest/granule/Ingest.getBucket()

Deprecated

@cmrjs/ingestConcept, instead use the CMR object methods. @cmrjs/CMR.ingestGranule or @cmrjs/CMR.ingestCollection
@cmrjs/searchConcept, instead use the CMR object methods. @cmrjs/CMR.searchGranules or @cmrjs/CMR.searchCollections
@cmrjs/deleteConcept, instead use the CMR object methods. @cmrjs/CMR.deleteGranule or @cmrjs/CMR.deleteCollection

Assets 2
Loading