Avoid accidental EC2 recreation

[cat5inthecradle]

We released a change to the BlockDeviceMapping that triggered recreation of the EC2 instance. This is not good, as the Daemon server in staging, test, etc

Links

• jira ticket: INF-497

Testing story

Deployment strategy

Follow-up work

Privacy

Security

Caching

PR Checklist:

• Tests provide adequate coverage
• Privacy and Security impacts have been assessed
• Code is well-commented
• New features are translatable or updates will not break translations
• Relevant documentation has been added or updated
• User impact is well-understood and desirable
• Pull Request is labeled appropriately
• Follow-up work items (including potential tech debt) are tracked and linked

[sureshc]

Should we set DisableApiTermination to true on the daemon EC2 Instance unless the environment == adhoc?

[sureshc]

Is there a mechanism we can use to take regular Snapshots of the daemon Volumes with Lifecycle management to prune the old Snapshots? AWS documentation indicates that it’s safest to stop an Instance before taking a Snapshot.

[cat5inthecradle]

Is there a mechanism we can use to take regular Snapshots of the daemon Volumes with Lifecycle management to prune the old Snapshots? AWS documentation indicates that it’s safest to stop an Instance before taking a Snapshot.

Definitely safest to stop it - is there a window of time where we can afford to do that? Live snapshots are better than no snapshots though.

When I've done this in the past we implemented a general cloudwatch event and lambda(s) that looked for tags on EC2 instances that defined the backup rules (backup: true, retain: 5 for example). But it looks like nowadays you can use Data Lifecycle Manager.