v9.0.5

Release v9.0.5

Important information

This release is only compatible with Cumulus v18.x.x and up.

• Full Change Comparison: v9.0.4...v9.0.5

Migration Notes

If you are deploying ORCA for the first time or migrating from v6, no changes are needed.

If you are currently on v8 or v9, this means you already have load balancer deployed and you need to delete the load balancer target group before deploying this version. This is because terraform cannot delete existing load balancer target groups having a listener attached. Adding a HTTPS to the target group requires replacing the target group. Once the target group is deleted, you should be able to deploy ORCA.

1. From AWS EC2 console, go to your load balancer named <prefix-gql-a> and select the Listeners and rules tab. Delete the rule.
2. Delete your target group <random_name>-gql-a. The target group name has been randomized to avoid terraform resource error.
3. Deploy ORCA.

If deployed correctly, the target group health checks should show as healthy.

Added

• ORCA-450 - Removed Access Control List (ACL) requirement and added BucketOwnerEnforced to ORCA bucket objects.
• ORCA-452 - Added Deny non SSL policy to S3 buckets in modules/dr_buckets/dr_buckets.tf and modules/dr_buckets_cloudformation/ dr-buckets.yaml

Changed

• ORCA-441 - Updated policies for ORCA buckets and copy_to_archive to give them only the permissions needed to restrict unwanted/unintended actions.
• ORCA-746 - Enabled HTTPS listener in application load balancer for GraphQL server using AWS Certificate Manager.
• ORCA-828 - Added prefix to ORCA SNS topic names to avoid object already exists errors.

Security

• ORCA-821 - Fixed snyk vulnerabilities from snyk report showing high issues and upgraded docusaurus to v3.1.0.

v9.0.4

Release v9.0.4

Important information

This release is only compatible with Cumulus v18.x.x and up.

• Full Change Comparison: v9.0.3...v9.0.4

Migration Notes

• For users upgrading from ORCA v8.x.x to v9.x.x, follow the below steps before deploying:
  1. Run the Lambda deletion script found in python3 bin/delete_lambda.py which will delete all of the ORCA lambdas with a provided prefix. You can also delete them manually in the AWS console.
  2. Navigate to the AWS console and search for the Cumulus RDS security group.
  3. Remove the inbound rule with the source of PREFIX-vpc-ingress-all-egress in Cumulus RDS security group.
  4. Search for PREFIX-vpc-ingress-all-egress and delete the security group NOTE: Due to the Lambdas using ENIs, when deleting the security groups it may say they are still associated with a Lambda that was deleted by the script. AWS may need a few minutes to refresh to fully disassociate the ENIs completely, if this error appears wait a few minutes and then try again.

Changed

• ORCA-826 - Changed bin/delete_lambda.py to delete ORCA lambdas based on their tags.
• ORCA-827 - Changed ORCA API gateway stage name from orca to orca_api to avoid confusion in the URL path. The new ORCA execute API URL will be https://<API_ID>.execute-api.<AWS_REGION>.amazonaws.com/orca_api.

Fixed

• ORCA-827 Fixed API gateway URL not found issue seen in ORCA v9.0.3.

v9.0.4-beta

Release v9.0.4-beta

v9.0.3

Release v9.0.3

Important information

🔥 This release is only compatible with Cumulus v18.x.x and up🔥

• Full Change Comparison: v9.0.2...v9.0.3

Migration notes

If you are migrating from ORCA v8.x.x to this version, see the migration notes under v9.0.0.

Fixed

• ORCA-823 Fixed ORCA security group related deployment error seen in ORCA v9.0.2.

v9.0.2

Release v9.0.2

Important information

🔥 This release is only compatible with Cumulus v18.x.x and up🔥

• Full Change Comparison: v9.0.1...v9.0.2

Added

• ORCA-366 Added unit test for shared libraries.
• ORCA-769 Added API Gateway Stage resource to modules/api-gateway/main.tf
• ORCA-369 Added DR S3 bucket template to modules/dr_buckets/dr_buckets.tf and updated S3 deployment documentation with steps.

Changed

• ORCA-784 Changed documentation to replace restore with copy based on task's naming as well as changed file name from website/docs/operator/restore-to-orca.mdx to website/docs/operator/reingest-to-orca.mdx.
• ORCA-724 Updated ORCA recovery documentation to include recovery workflow process and relevant inputs and outputs in website/docs/operator/data-recovery.md.
• ORCA-789 Updated extract_filepaths_for_granule to more flexibly match file-regex values to keys.
• ORCA-787 Modified modules/api-gateway/main.tf api gateway stage name to remove the extra orca from the data management URL path
• ORCA-805 Changed modules/security_groups/main.tf security group resource name from vpc_postgres_ingress_all_egress to vpc-postgres-ingress-all-egress to resolve errors when upgrading from ORCA v8 to v9. Also removed graphql_1 dependency module.orca_lambdas since this module does not depend on the lambda module in modules/orca/main.tf

Removed

• ORCA-361 Removed hardcoded test values from extract_file_paths_for_granule unit tests.
• ORCA-710 Removed duplicate logging messages in integration_test/workflow_tests/custom_logger.py
• ORCA-815 Removed steps for creating buckets using NGAP form in ORCA archive bucket documentation.

Fixed

• ORCA-811 Fixed cumulus_orca docker image by updating nodejs installation process.
• ORCA-802 Fixed extract_file_for_granule documentation and schemas to include collectionId in input.
• ORCA-785 Fixed checksum integrity issue in ORCA documentation bamboo pipeline.
• ORCA-820 Updated bandit and moto libraries to fix some snyk vulnerabilities.

v9.0.1

Release v9.0.1

Important information

🔥 This release is only compatible with Cumulus v18.x.x and up🔥

• Full Change Comparison: v9.0.0...v9.0.1

Added

• ORCA-766 Created AWS cloudformation template that can be used to deploy ORCA DR buckets.
• ORCA-765 Updated ORCA "Creating the Glacier Bucket" documentation with instructions to deploy ORCA DR buckets using cloudformation.

Changed

• ORCA-780 Updated ORCA "Deployment with Cumulus" documentation with instructions and examples to run ORCA recovery and archive workflows.
• ORCA-704 Updated dr-buckets.tf.template and buckets.tf.template with provider block to deploy in the us-west-2 region due to deployments failing in the other regions.
• ORCA-708 Updated integration_test/shared/setup-orca.sh script to use the root folder instead of cloning in a duplicate repository.

Fixed

• ORCA-731 Updated boto3 library used for unit tests to version 1.28.76 from version 1.18.40 to fix unit test warnings.

Security

• ORCA-778 Upgraded Docusaurus to version 2.4.3 to fix snyk vulnerabilities and security issues.
• ORCA-737 Updated moto library used for unit tests to version 4.2.2 from version 2.0.

v9.0.0

Important information

🔥 This release is only compatible with Cumulus v18.x.x and up🔥

• Full Change Comparison: v8.1.0...v9.0.0

Migration notes

Update teraform to version 1.5.x

Security

• ORCA-729 Updated terraform provider to use the latest version 1.5
• ORCA-713 Updated terraform, Dockerfile, and other IAC elements for best practices and security where able.

v6.0.4

Important Information

This release is only compatible with Cumulus versions 15.x and 16.x

• Full Change Log available at https://github.com/nasa/cumulus-orca/blob/release-6.0.4/CHANGELOG.md
• Full Change Comparison: v6.0.3...v6.0.4

Fixed

• ORCA-738 Update cumulus-process to v1.2.0 and cumulus-message-adapter-python to v2.1.0. To alleviate potential issues related to timeouts when using CMA calls.

v8.1.0

Important information

🔥 This release is only compatible with Cumulus v17.x.x 🔥

• Full Change Log available at https://github.com/nasa/cumulus-orca/blob/release-8.1.0/CHANGELOG.md
• Full Change Comparison: v6.0.3...v8.1.0

Migration Notes

• If utilizing the copied_to_glacier output property of copy_to_glacier,
  rename to new key copied_to_orca.

• If utilizing the orca_lambda_copy_to_glacier_arn output of Terraform, likely as a means of pulling the lambda into your workflows,
  rename to new key orca_lambda_copy_to_archive_arn

• If utilizing the orca_lambda_request_files_arn output of Terraform, likely as a means of pulling the lambda into your workflows, rename to new key orca_lambda_request_from_archive_arn

• If desired, use the optional recoveryBucketOverride property in extract_filepaths_for_granule input schema to override the default recovery bucket. See example below.

  {
    "input":
      {
        "granules": [
          {
            "granuleId": "MOD09GQ.A0219114.N5aUCG.006.0656338553321",
            "recoveryBucketOverride": "<YOUR_RECOVERY_BUCKET>",
            "files": [
              {
                "key": "MOD09GQ___006/2017/MOD/MOD09GQ.A0219114.N5aUCG.006.  0656338553321.h5",
                "bucket": "cumulus-test-sandbox-protected",
                "fileName": "MOD09GQ.A0219114.N5aUCG.006.0656338553321.h5",
              }
            ]
          }
        ]
    }
  }

• If utilizing the output of the OrcaRecoveryWorkflow, adjust to the simplified output schema. See example below:

  {
      "granules": [
      {
        "granuleId": "integrationGranuleId",
        "keys": [
          {
            "key": "PODAAC/SWOT/ancillary_data_input_forcing_ECCO_V4r4.tar.gz",
            "destBucket": "PREFIX-public"
          }
        ],
        "recoverFiles": [
          {
            "success": true,
            "filename": "ancillary_data_input_forcing_ECCO_V4r4.tar.gz",
            "keyPath": "PODAAC/SWOT/ancillary_data_input_forcing_ECCO_V4r4.tar.  gz",
            "restoreDestination": "PREFIX-public",
            "s3MultipartChunksizeMb": null,
            "statusId": 1,
            "requestTime": "2023-02-10T21:06:13.071287+00:00",
            "lastUpdate": "2023-02-10T21:06:13.071287+00:00"
          }
        ]
      }
    ],
    "asyncOperationId": "770a85f2-f933-4440-90b5-1a8039557538"
  }

• The output format of copy_to_archive lambda and step-function has been simplified. If accessing these resources outside of a Cumulus perspective, instead of accessing output["payload"]["granules"] you now use output["granules"].

• Cumulus is not currently compatible with the changes to copy_to_archive.

  • This section will be updated when a compatible version is created.
  • deployment-with-cumulus.md will also be updated.
  • copy_to_archive_adapter/README.md will also be updated.
  • restore-to-orca.mdx will also be updated.

• Cumulus is not currently compatible with the changes to the Recovery Workflow step-function.

  • This section will be updated when a compatible version is created.
  • deployment-with-cumulus.md will also be updated.
  • orca_recovery_adapter/README.md will also be updated.

• Update the bucket policy for your system-bucket to allow load balancer to post server access logs to the bucket. See the instructions here.

• InternalReconcileReport Phantom and Mismatch reports are now available via GraphQL.

  • API Gateway access is now deprecated, and will be removed in a future update.
  • Use the orca_graphql_load_balancer_dns_name variable to send your queries to GraphQL as json strings in a POST request.

• Users will need to update their orca-user password. The password must have the following requirements otherwise the db_deploy lambda will fail during deployment.

  • one upper case letter
  • one lower case letter
  • one digit
  • one special character
  • minimum length of 12

  Update db_user_password variable in your cumulus-tf/terraform.tfvars file to match the new password requirement and then run terraform. db_deploy lambda will automatically update your new password.

• Changes have been made to SQS message processing that are not backwards compatible. Halt ingest and wait for the PREFIX-orca-status-update-queue.fifo queue to empty before applying update.

  • If the queue is stuck or becomes stuck, it may be necessary to flush the queue and its associated Dead Letter Queue.

• The input format of the ORCA Recovery Workflow step-function has been modified.
  If accessing these resources outside of a Cumulus perspective, go to orca_recover_workflow.asl.json and look at config elements to see the new paths. Additionally, add a collectionId property to each granule passed in.

• collectionId properties have been added to Recovery Jobs and Recovery Granules API.

  • For Recovery Jobs, it is only added to output.
  • For Recovery Granules, it is now required on input and will be returned on output.

• Update the orca.tf file to include aws_region. See example below.

  ## ORCA Module
  ## =============================================================================
  module "orca" {
    source = "https://github.com/nasa/cumulus-orca/releases/download/v6.0.0/cumulus-orca-terraform.zip//modules"
  ## --------------------------
  ## Cumulus Variables
  ## --------------------------
  ## REQUIRED
  aws_region               = var.region
  buckets                  = var.buckets
  lambda_subnet_ids        = var.lambda_subnet_ids
  permissions_boundary_arn = var.permissions_boundary_arn
  prefix                   = var.prefix
  system_bucket            = var.system_bucket
  vpc_id                   = var.vpc_id
  workflow_config          = module.cumulus.workflow_config
  
  ## OPTIONAL
  tags        = local.tags
  
  ## --------------------------
  ## ORCA Variables
  ## --------------------------
  ## REQUIRED
  db_admin_password        = var.db_admin_password
  db_user_password         = var.db_user_password
  db_host_endpoint         = var.db_host_endpoint
  dlq_subscription_email   = var.dlq_subscription_email
  orca_default_bucket      = var.orca_default_bucket
  orca_reports_bucket_name = var.orca_reports_bucket_name
  rds_security_group_id    = var.rds_security_group_id
  s3_access_key            = var.s3_access_key
  s3_secret_key            = var.s3_secret_key
  
  ## OPTIONAL
  db_admin_username                                    = "postgres"
  default_multipart_chunksize_mb                       = 250
  internal_report_queue_message_retention_time_seconds = 432000
  orca_default_recovery_type                           = "Standard"
  orca_default_storage_class                           = "GLACIER"
  orca_delete_old_reconcile_jobs_frequency_cron        = "cron(0 0 ? * SUN *)"
  orca_ingest_lambda_memory_size                       = 2240
  orca_ingest_lambda_timeout                           = 720
  orca_internal_reconciliation_expiration_days         = 30
  orca_recovery_buckets                                = []
  orca_recovery_complete_filter_prefix                 = ""
  orca_recovery_expiration_days                        = 5
  orca_recovery_lambda_memory_size                     = 128
  orca_recovery_lambda_timeout                         = 720
  orca_recovery_retry_limit                            = 3
  orca_recovery_retry_interval                         = 1
  orca_recovery_retry_backoff                          = 2
  s3_inventory_queue_message_retention_time_seconds    = 432000
  s3_report_frequency                                  = "Daily"
  sqs_delay_time_seconds                               = 0
  sqs_maximum_message_size                             = 262144
  staged_recovery_queue_message_retention_time_seconds = 432000
  status_update_queue_message_retention_time_seconds   = 777600
  vpc_endpoint_id                                      = null
  }

v9.0.0-beta

Release v9.0.0-beta

Update teraform to version 1.5.x