Skip to content

v8.1.0
Compare
Choose a tag to compare
@bhazuka bhazuka released this 23 Aug 18:58
· 264 commits to master since this release
v8.1.0
0df4f1c
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Important information

🔥 This release is only compatible with Cumulus v17.x.x 🔥

Migration Notes

  • If utilizing the copied_to_glacier output property of copy_to_glacier,
    rename to new key copied_to_orca.

  • If utilizing the orca_lambda_copy_to_glacier_arn output of Terraform, likely as a means of pulling the lambda into your workflows,
    rename to new key orca_lambda_copy_to_archive_arn

  • If utilizing the orca_lambda_request_files_arn output of Terraform, likely as a means of pulling the lambda into your workflows, rename to new key orca_lambda_request_from_archive_arn

  • If desired, use the optional recoveryBucketOverride property in extract_filepaths_for_granule input schema to override the default recovery bucket. See example below.

    {
  "input":
    {
      "granules": [
        {
          "granuleId": "MOD09GQ.A0219114.N5aUCG.006.0656338553321",
          "recoveryBucketOverride": "<YOUR_RECOVERY_BUCKET>",
          "files": [
            {
              "key": "MOD09GQ___006/2017/MOD/MOD09GQ.A0219114.N5aUCG.006.  0656338553321.h5",
              "bucket": "cumulus-test-sandbox-protected",
              "fileName": "MOD09GQ.A0219114.N5aUCG.006.0656338553321.h5",
            }
          ]
        }
      ]
  }
}

  • If utilizing the output of the OrcaRecoveryWorkflow, adjust to the simplified output schema. See example below:

    {
    "granules": [
    {
      "granuleId": "integrationGranuleId",
      "keys": [
        {
          "key": "PODAAC/SWOT/ancillary_data_input_forcing_ECCO_V4r4.tar.gz",
          "destBucket": "PREFIX-public"
        }
      ],
      "recoverFiles": [
        {
          "success": true,
          "filename": "ancillary_data_input_forcing_ECCO_V4r4.tar.gz",
          "keyPath": "PODAAC/SWOT/ancillary_data_input_forcing_ECCO_V4r4.tar.  gz",
          "restoreDestination": "PREFIX-public",
          "s3MultipartChunksizeMb": null,
          "statusId": 1,
          "requestTime": "2023-02-10T21:06:13.071287+00:00",
          "lastUpdate": "2023-02-10T21:06:13.071287+00:00"
        }
      ]
    }
  ],
  "asyncOperationId": "770a85f2-f933-4440-90b5-1a8039557538"
}

  • The output format of copy_to_archive lambda and step-function has been simplified. If accessing these resources outside of a Cumulus perspective, instead of accessing output["payload"]["granules"] you now use output["granules"].

  • Cumulus is not currently compatible with the changes to copy_to_archive.

    • This section will be updated when a compatible version is created.
    • deployment-with-cumulus.md will also be updated.
    • copy_to_archive_adapter/README.md will also be updated.
    • restore-to-orca.mdx will also be updated.

  • Cumulus is not currently compatible with the changes to the Recovery Workflow step-function.

    • This section will be updated when a compatible version is created.
    • deployment-with-cumulus.md will also be updated.
    • orca_recovery_adapter/README.md will also be updated.

  • Update the bucket policy for your system-bucket to allow load balancer to post server access logs to the bucket. See the instructions here.

  • InternalReconcileReport Phantom and Mismatch reports are now available via GraphQL.

    • API Gateway access is now deprecated, and will be removed in a future update.
    • Use the orca_graphql_load_balancer_dns_name variable to send your queries to GraphQL as json strings in a POST request.

  • Users will need to update their orca-user password. The password must have the following requirements otherwise the db_deploy lambda will fail during deployment.

    • one upper case letter
    • one lower case letter
    • one digit
    • one special character
    • minimum length of 12

    Update db_user_password variable in your cumulus-tf/terraform.tfvars file to match the new password requirement and then run terraform. db_deploy lambda will automatically update your new password.

  • Changes have been made to SQS message processing that are not backwards compatible. Halt ingest and wait for the PREFIX-orca-status-update-queue.fifo queue to empty before applying update.

    • If the queue is stuck or becomes stuck, it may be necessary to flush the queue and its associated Dead Letter Queue.

  • The input format of the ORCA Recovery Workflow step-function has been modified.
    If accessing these resources outside of a Cumulus perspective, go to orca_recover_workflow.asl.json and look at config elements to see the new paths. Additionally, add a collectionId property to each granule passed in.

  • collectionId properties have been added to Recovery Jobs and Recovery Granules API.

    • For Recovery Jobs, it is only added to output.
    • For Recovery Granules, it is now required on input and will be returned on output.

  • Update the orca.tf file to include aws_region. See example below.

    ## ORCA Module
## =============================================================================
module "orca" {
  source = "https://github.com/nasa/cumulus-orca/releases/download/v6.0.0/cumulus-orca-terraform.zip//modules"
## --------------------------
## Cumulus Variables
## --------------------------
## REQUIRED
aws_region               = var.region
buckets                  = var.buckets
lambda_subnet_ids        = var.lambda_subnet_ids
permissions_boundary_arn = var.permissions_boundary_arn
prefix                   = var.prefix
system_bucket            = var.system_bucket
vpc_id                   = var.vpc_id
workflow_config          = module.cumulus.workflow_config

## OPTIONAL
tags        = local.tags

## --------------------------
## ORCA Variables
## --------------------------
## REQUIRED
db_admin_password        = var.db_admin_password
db_user_password         = var.db_user_password
db_host_endpoint         = var.db_host_endpoint
dlq_subscription_email   = var.dlq_subscription_email
orca_default_bucket      = var.orca_default_bucket
orca_reports_bucket_name = var.orca_reports_bucket_name
rds_security_group_id    = var.rds_security_group_id
s3_access_key            = var.s3_access_key
s3_secret_key            = var.s3_secret_key

## OPTIONAL
db_admin_username                                    = "postgres"
default_multipart_chunksize_mb                       = 250
internal_report_queue_message_retention_time_seconds = 432000
orca_default_recovery_type                           = "Standard"
orca_default_storage_class                           = "GLACIER"
orca_delete_old_reconcile_jobs_frequency_cron        = "cron(0 0 ? * SUN *)"
orca_ingest_lambda_memory_size                       = 2240
orca_ingest_lambda_timeout                           = 720
orca_internal_reconciliation_expiration_days         = 30
orca_recovery_buckets                                = []
orca_recovery_complete_filter_prefix                 = ""
orca_recovery_expiration_days                        = 5
orca_recovery_lambda_memory_size                     = 128
orca_recovery_lambda_timeout                         = 720
orca_recovery_retry_limit                            = 3
orca_recovery_retry_interval                         = 1
orca_recovery_retry_backoff                          = 2
s3_inventory_queue_message_retention_time_seconds    = 432000
s3_report_frequency                                  = "Daily"
sqs_delay_time_seconds                               = 0
sqs_maximum_message_size                             = 262144
staged_recovery_queue_message_retention_time_seconds = 432000
status_update_queue_message_retention_time_seconds   = 777600
vpc_endpoint_id                                      = null
}
Assets 3