v1.13.0
Notable Changes
- Separate DB stack for the data layer (Dynamo and Elasticsearch)
- All stacks use the
app/config.yml
- ECS Autoscaling
- Default AMI is NASA NGAP AMI
- Elasticsearch is deployed behind the VPC
- Support for configuring private API Gateways
Migration Steps
- Read through the notes below, particularly noting the breaking changes
- Delete your Cumulus
app
stack in Cloudformation.- If running Cumulus within a VPC and extended downtime is acceptable, we recommend doing this at the end of the day to allow AWS backend resources and network interfaces to be cleaned up overnight.
- Setup a Service Linked Role for Elasticsearch
- This is a one-time setup for the account. You can follow the instructions to use the AWS console or AWS CLI here or use the following AWS CLI command:
aws iam create-service-linked-role --aws-service-name es.amazonaws.com
- This is a one-time setup for the account. You can follow the instructions to use the AWS console or AWS CLI here or use the following AWS CLI command:
- Delete your
iam/config.yml
and move your configurations toapp/config.yml
See notes below for CUMULUS-1310. - Redeploy your
iam
stack - Configure your desired security group using an existing security group (In an NGAP environment, this would be the default application security group) in your
app/config.yml
, populatevpc.securityGroup
- Configure your API to be private if desired using the
apiConfigs
configuration in yourapp/config.yml
- This is needed for NGAP compliance. Please read this wiki page before deploying to make sure you have the prerequisites. This can be done at a later time.
- If not in an NGAP environment, configure your
ecs.amiid
in yourapp/config.yml
to point to an AWS AMI. - Deploy your
db
stack - Deploy your
app
stack
PLEASE NOTE
CUMULUS-802 added some additional IAM permissions to support ECS autoscaling and changes were needed to run all lambdas in the VPC, so you will have to redeploy your IAM stack.
As a result of the changes for CUMULUS-1193, CUMULUS-1264, and CUMULUS-1310, you must delete your existing stacks (except IAM) before deploying this version of Cumulus.
If running Cumulus within a VPC and extended downtime is acceptable, we recommend doing this at the end of the day to allow AWS backend resources and network interfaces to be cleaned up overnight.
BREAKING CHANGES
-
CUMULUS-1228
- The default AMI used by ECS instances is now an NGAP-compliant AMI. This
will be a breaking change for non-NGAP deployments. If you do not deploy to
NGAP, you will need to find the AMI ID of the
most recent Amazon ECS-optimized AMI,
and set theecs.amiid
property in your config. Instructions for finding
the most recent NGAP AMI can be found using
these instructions.
- The default AMI used by ECS instances is now an NGAP-compliant AMI. This
-
CUMULUS-1310
- Database resources (DynamoDB, ElasticSearch) have been moved to an independent
db
stack.
Migrations for this version will need to be user-managed. (e.g. elasticsearch and dynamoDB).
Order of stack deployment isiam
->db
->app
. - All stacks can now be deployed using a single
config.yml
file, i.e.:kes cf deploy --kes-folder app --template node_modules/@cumulus/deployment/[iam|db|app] [...]
Backwards-compatible. For development, please re-runnpm run bootstrap
to build newkes
overrides.
Deployment docs have been updated to show how to deploy a single-config Cumulus instance. params
have been moved: Nestparams
fields underapp
,db
oriam
to override all Parameters for a particular stack's cloudformation template. Backwards-compatible with multi-config setups.stackName
andstackNameNoDash
have been retired. Useprefix
andprefixNoDash
instead.- The
iams
section inapp/config.yml
IAM roles has been deprecated as a user-facing parameter,
unless your IAM role ARNs do not match the convention shown in@cumulus/deployment/app/config.yml
- The
vpc.securityGroup
will need to be set with a pre-existing security group ID to use Cumulus in a VPC. Must allow inbound HTTP(S) (Port 443).
- Database resources (DynamoDB, ElasticSearch) have been moved to an independent
-
CUMULUS-1212
@cumulus/post-to-cmr
will now fail if any granules being processed are missing a metadata file. You can set the new config optionskipMetaCheck
totrue
to pass post-to-cmr without a metadata file.
-
CUMULUS-1232
@cumulus/sync-granule
will no longer silently pass if no checksum data is provided. It will use input
from the granule object to:- Verify checksum if
checksumType
andchecksumValue
are in the file record OR a checksum file is provided
(throwsInvalidChecksum
on fail), else log warning that no checksum is available. - Then, verify synced S3 file size if
file.size
is in the file record (throwsUnexpectedFileSize
on fail),
else log warning that no file size is available. - Pass the step.
- Verify checksum if
-
CUMULUS-1264
- The Cloudformation templating and deployment configuration has been substantially refactored.
CumulusApiDefault
nested stack resource has been renamed toCumulusApiDistribution
CumulusApiV1
nested stack resource has been renamed toCumulusApiBackend
- The
urs: true
config option for when defining your lambdas (e.g. inlambdas.yml
) has been deprecated. There are two new options to replace it:urs_redirect: 'token'
: This will expose aTOKEN_REDIRECT_ENDPOINT
environment variable to your lambda that references the/token
endpoint on the Cumulus backend APIurs_redirect: 'distribution'
: This will expose aDISTRIBUTION_REDIRECT_ENDPOINT
environment variable to your lambda that references the/redirect
endpoint on the Cumulus distribution API
- The Cloudformation templating and deployment configuration has been substantially refactored.
-
CUMULUS-1193
- The elasticsearch instance is moved behind the VPC.
- Your account will need an Elasticsearch Service Linked role. This is a one-time setup for the account. You can follow the instructions to use the AWS console or AWS CLI here or use the following AWS CLI command:
aws iam create-service-linked-role --aws-service-name es.amazonaws.com
-
CUMULUS-802
- ECS
maxInstances
must be greater thanminInstances
. If you use defaults, no change is required.
- ECS
-
CUMULUS-1269
- Brought Cumulus data models in line with CNM JSON schema:
- Renamed file object
fileType
field totype
- Renamed file object
fileSize
field tosize
- Renamed file object
checksumValue
field tochecksum
where not already done. - Added
ancillary
andlinkage
type support to file objects.
- Renamed file object
- Brought Cumulus data models in line with CNM JSON schema:
Added
-
CUMULUS-1242 - Added
sqs2sfThrottle
lambda. The lambda reads SQS messages for queued executions and uses semaphores to only start new executions if the maximum number of executions defined for the priority key (cumulus_meta.priorityKey
) has not been reached. Any SQS messages that are read but not used to start executions remain in the queue. -
CUMULUS-1240
- Added
sfSemaphoreDown
lambda. This lambda receives SNS messages and for each message it decrements the semaphore used to track the number of running executions if:- the message is for a completed/failed workflow AND
- the message contains a level of priority (
cumulus_meta.priorityKey
)
- Added
sfSemaphoreDown
lambda as a subscriber to thesfTracker
SNS topic
- Added
-
CUMULUS-1265
- Added
apiConfigs
configuration option to configure API Gateway to be private - All internal lambdas configured to run inside the VPC by default
- Removed references to
NoVpc
lambdas from documentation andexample
folder.
- Added
-
CUMULUS-802
- Adds autoscaling of ECS clusters
- Adds autoscaling of ECS services that are handling StepFunction activities
- See ECS deployment configuration options
Changed
-
Updated
@cumulus/ingest/http/httpMixin.list()
to trim trailing spaces on discovered filenames -
CUMULUS-1310
- Database resources (DynamoDB, ElasticSearch) have been moved to an independent
db
stack.
This will enable future updates to avoid affecting database resources or requiring migrations.
Migrations for this version will need to be user-managed.
(e.g. elasticsearch and dynamoDB).
Order of stack deployment isiam
->db
->app
. - All stacks can now be deployed using a single
config.yml
file, i.e.:kes cf deploy --kes-folder app --template node_modules/@cumulus/deployment/[iam|db|app] [...]
Backwards-compatible. Please re-runnpm run bootstrap
to build newkes
overrides.
Deployment docs have been updated to show how to deploy a single-config Cumulus instance. params
fields should now be nested under the stack key (i.e.app
,db
oriam
) to provide Parameters for a particular stack's cloudformation template,
for use with single-config instances. Keys must match the name of the deployment package folder (app
,db
, oriam
).
Backwards-compatible with multi-config setups.stackName
andstackNameNoDash
have been retired as user-facing config parameters. Useprefix
andprefixNoDash
instead.
This will be used to create stack names for all stacks in a single-config use case.
stackName
may still be used as an override in multi-config usage, although this is discouraged.
Warning: overriding thedb
stack'sstackName
will require you to setdbStackName
in yourapp/config.yml
.
This parameter is required to fetch outputs from thedb
stack to reference in theapp
stack.- The
iams
section inapp/config.yml
IAM roles has been retired as a user-facing parameter,
unless your IAM role ARNs do not match the convention shown in@cumulus/deployment/app/config.yml
In that case, overridingiams
in your own config is recommended. iam
anddb
cloudformation.yml
file names will have respective prefixes (e.giam.cloudformation.yml
).- Cumulus will now only attempt to create reconciliation reports for buckets of the
private
,public
andprotected
types. - Cumulus will no longer set up its own security group.
To pass a pre-existing security group for in-VPC deployments as a parameter to the Cumulus template, populatevpc.securityGroup
inconfig.yml
.
This security group must allow inbound HTTP(S) traffic (Port 443). SSH traffic (Port 22) must be permitted for SSH access to ECS instances. - Deployment docs have been updated with examples for the new deployment model.
- Database resources (DynamoDB, ElasticSearch) have been moved to an independent
-
CUMULUS-1236
- Moves access to public files behind the distribution endpoint. Authentication is not required, but direct http access has been disallowed.
-
CUMULUS-1223
- Adds unauthenticated access for public bucket files to the Distribution API. Public files should be requested the same way as protected files, but for public files a redirect to a self-signed S3 URL will happen without requiring authentication with Earthdata login.
-
CUMULUS-1232
- Unifies duplicate handling in
ingest/granule.handleDuplicateFile
for maintainability. - Changed
ingest/granule.ingestFile
andmove-granules/index.moveFileRequest
to use new function. - Moved file versioning code to
ingest/granule.moveGranuleFileWithVersioning
ingest/granule.verifyFile
now also testsfile.size
for verification if it is in the file record and throws
UnexpectedFileSize
error for file size not matching input.ingest/granule.verifyFile
logs warnings if checksum and/or file size are not available.
- Unifies duplicate handling in
-
CUMULUS-1193
- Moved reindex CLI functionality to an API endpoint. See API docs
-
CUMULUS-1207
- No longer disable lambda event source mappings when disabling a rule
Fixed
-
Updated Lerna publish script so that published Cumulus packages will pin their dependencies on other Cumulus packages to exact versions (e.g.
1.12.1
instead of^1.12.1
) -
CUMULUS-1203
- Fixes IAM template's use of intrinsic functions such that IAM template overrides now work with kes
-
CUMULUS-1268
- Deployment will not fail if there are no ES alarms or ECS services