Introducing allFilesPresent metadata field for Cumulus Collections. Fixes Issue #3193

[micahThor]

Summary: Introducing allFilesPresent metadata field for Cumulus Collections

Addresses CUMULUS-3193: Introducing allFilesPresent metadata field for Cumulus Collections(#3193)

Changes

Change allows Cumulus Collections to enforce that all files specified in Collections list to be present before ingesting Granules.

Build and Tests

Successful Build

webpack 5.75.0 compiled with 9 warnings in 19512 ms
lerna success run Ran npm script 'build' in 28 packages in 107.9s:
lerna success - @cumulus/ftp-populate-test-lambda
lerna success - @cumulus/python-process-activity
lerna success - @cumulus/python-reference-activity
lerna success - @cumulus/python-reference-task
lerna success - @cumulus/example-lib
lerna success - @cumulus/postgres-migration-async-operation
lerna success - @cumulus/api
lerna success - @cumulus/ingest
lerna success - @cumulus/integration-tests
lerna success - @cumulus/s3-credentials-endpoint
lerna success - @cumulus/discover-granules
lerna success - @cumulus/discover-pdrs
lerna success - @cumulus/files-to-granules
lerna success - @cumulus/hello-world
lerna success - @cumulus/hyrax-metadata-updates
lerna success - @cumulus/move-granules
lerna success - @cumulus/parse-pdr
lerna success - @cumulus/pdr-status-check
lerna success - @cumulus/post-to-cmr
lerna success - @cumulus/queue-granules
lerna success - @cumulus/queue-pdrs
lerna success - @cumulus/queue-workflow
lerna success - @cumulus/sf-sqs-report
lerna success - @cumulus/sync-granule
lerna success - @cumulus/test-processing
lerna success - @cumulus/update-cmr-access-constraints
lerna success - @cumulus/update-granules-cmr-metadata-file-links
lerna success - @cumulus/cumulus-test-cleanup

Successful Tests

lerna info run Ran npm script 'test' in '@cumulus/discover-granules' in 8.8s:
> @cumulus/discover-granules@13.4.0 test
> ../../node_modules/.bin/ava
  ✔ discover granules uses default concurrency of 3
  ✔ discover granules uses configured concurrency
  ✔ discover granules sets the GRANULES environment variable and logs the granules
  ✔ handleDuplicates throws Error on duplicateHandling set to "error"
  ✔ handleDuplicates filters on duplicateHandling set to "skip"
  ✔ handleDuplicates does not filter when duplicateHandling is set to "replace"
  ✔ handleDuplicates throws Error on an invalid duplicateHandling configuration
  ✔ filterDuplicates returns a set of filtered keys
  ✔ handleDuplicates does not filter when duplicateHandling is set to "version"
  ✔ checkGranuleHasNoDuplicate returns false when API lambda returns a granule
  ✔ checkGranuleHasNoDuplicate returns a granuleId string when the API lambda returns a 404/Not Found error
  ✔ checkGranuleHasNoDuplicate throws an error if the API lambda throws an error other than 404/Not Found
  ✔ checkGranuleHasNoDuplicate throws an error when API lambda returns a granule and duplicateHandling is set to "error"
  ✔ checkGranuleHasNoDuplicate throws an error on an unexpected API lambda return
  ✔ discover granules sets the correct dataType for granules (229ms)
  ✔ discover granules using HTTP (233ms)
  ✔ discover granules using FTP (475ms)
  ✔ discover granules using SFTP (1.2s)
  ✔ discover granules without collection files config using S3 (5.3s)
  ✔ discover granules with allFilesPresent set to true with files missing (5.4s)
  ✔ discover granules with allFilesPresent set to true (5.4s)
  ✔ discover granules with allFilesPresent set to true with extra files for granule (5.4s)
  ✔ discover granules without collection files config, but configuring task to ignore it and overriding collection config not to ignore it, using S3 (5.5s)
  ✔ discover granules using S3 (5.5s)
  ✔ discover granules without collection files config, but configuring collection to ignore it, using S3 (5.5s)
  ✔ discover granules without collection files config, but configuring task to ignore it, using S3 (5.5s)
  ✔ discover granules without collection files config for .nc files using S3 (5.6s)
  ✔ discover granules using S3 throws error when discovery fails (5.6s)
  ─
  28 tests passed

PR Checklist

• Update CHANGELOG
• Unit tests
• Ad-hoc testing - Deploy changes and test manually
• Integration tests

[micahThor]

Hello, can I get some help with the above failures? My builds and tests passed and I'm not sure which tests are failing here or why.

[botanical]

Hello, can I get some help with the above failures? My builds and tests passed and I'm not sure which tests are failing here or why.

@micahThor The build and test failures are to be expected and not a reflection of your work necessarily. I believe the builds and tests are failing because you don't have the correct permissions to this repo. We'll have someone on our team take a look at your PR as soon as possible. Thanks for the PR!

[botanical]

Hi @micahThor, thanks for the contribution! 🙇 Apologies for the delay in review. If you could also update these docs https://github.com/nasa/cumulus/blob/master/docs/workflow_tasks/discover_granules.md#task-inputs to include the new config field allFilesPresent, that would be great!