VDR/VER treatment of vulnerabilities in reusable EC2 AMIs

[Scott-Beauregard]

We're looking for clarification on how the VDR and VER rules apply to vulnerabilities in internally maintained Amazon Machine Images (AMIs), particularly when those images are used to provision production resources for multiple federal agencies.

Example environment

Consider a Rev. 5 Class C SaaS offering that uses internally maintained Windows Server AMIs to provision customer-specific EC2 web servers.

The AMIs are periodically rebuilt using newer Windows Server base images, but some installed application components remain at older patch levels. Newly provisioned servers subsequently receive the missing security updates through WSUS, generally within a few days of deployment.

The AMIs themselves are immutable, do not execute application workloads, and do not store or process federal customer data while stored. However, they are actively used to provision production servers that do process agency data.

Under the traditional Rev. 5 vulnerability scanning guidance, these internally maintained AMIs were not explicitly identified as separate scanning targets, unlike operating systems, databases, web applications, and container images.

We're trying to understand how the broader VDR/VER requirements apply to these reusable deployment artifacts.

Questions

1. Detection scope: Should an internally maintained EC2 AMI be treated as a separate, non-drifting information resource under VDR-TFR-PCD, subject to periodic vulnerability detection independently of the EC2 instances launched from it? Or can detection performed against deployed instances satisfy the applicable requirements for the underlying image?

2. Vulnerability evaluation: When a CVE affects software installed in the AMI, should the vulnerability be evaluated separately for the stored image, even though the vulnerable software is not executing? Or should the evaluation consider the actual exploitation conditions of the production instances created from the image?

3. Potential Agency Impact: If a common AMI is used to deploy independently operated, tenant-isolated environments for multiple federal agencies, should its potential agency impact be evaluated against the cumulative consequences across those agencies? Or should potential agency impact be determined from the realistic exploitation scenarios affecting the individual deployed environments?

4. Grouping and remediation: Under VER-EVA-GRV, can vulnerabilities in the source AMI and its deployed instances be managed as a consolidated vulnerability group, provided that differences in exploitability and agency impact are documented? If the deployed instances are patched but the source AMI remains vulnerable, does the continued existence and use of the image prevent the grouped vulnerability from being considered remediated?

We recognize that the VDR rules encourage secure deployment practices, detection after changes, and avoidance of known exploited vulnerabilities. Our question concerns the unit of vulnerability evaluation and reporting for reusable machine images, rather than whether maintaining updated images is a desirable security practice.

Clarification would help CSPs distinguish vulnerabilities in dormant deployment artifacts from vulnerabilities in active workloads, avoid duplicative reporting, and accurately represent the potential impact to federal agencies.

[pete-gov]

I love this scenario because it shows excellent security architecture decisions that the VDR is intended to reward. :)

I'm going to do my best to answer these questions based on the information supplied, but please remember that there is no substitute for technical security expertise that has deep familiarity with your specific architecture and processes. I may make assumptions that do not apply in your specific environment, or there may details I don't understand that render my responses moot. These responses are intended only to be illustrative about potential approach and are not formal indications of what you should or shouldn't do - it is entirely up to the cloud service provider to take responsibility for their approach based on their unique circumstances.

Detection scope: Should an internally maintained EC2 AMI be treated as a separate, non-drifting information resource under VDR-TFR-PCD, subject to periodic vulnerability detection independently of the EC2 instances launched from it? Or can detection performed against deployed instances satisfy the applicable requirements for the underlying image?

In the circumstances you've described, the immutable AMIs might reasonably be evaluated as information resources that are not likely to drift. They also might be grouped together by version, and if there are sufficient controls in place to ensure they have not been modified and that they must still be in their immutable state you could run various simple mechanisms to simply ensure that's true without having to go deep into all of them (and ideally you have a mechanism that already would alert if something was changed somehow). And you could do that with sampling if you can be entirely confident that will be accurate within a group. And vulnerability detection might not even need a scanner or similar, since you can simply run a comparison against a known manifest of installed software and configurations if they are truly immutable such that nothing new can be installed or run.

The nuance of all of this depends on how they are implemented and how truly immutable things are and how you can prove that, but there is certainly a world where you can simply not even have to do anything on a monthly cadence because mechanisms are in place to perform vulnerability detection continuously without impacting any of those running instances, just based on manifests/etc. Or you might test samples, or you might test all of them, depending on the likelihood of drift.

Whether or not you need to test deployed instances or would test deployed instances instead of the AMI depends entirely on the level of certainty you have that they can't drift. Ideally though I think you'd want to test the AMI based on the manifest and not do any scanning of a running system at all because you can programmatically alert if a CVE/etc. matching your manifest is released instead of waiting to do some silly scan.

Vulnerability evaluation: When a CVE affects software installed in the AMI, should the vulnerability be evaluated separately for the stored image, even though the vulnerable software is not executing? Or should the evaluation consider the actual exploitation conditions of the production instances created from the image?

I think the nuance of your implementation here matters a lot, along with the specific vulnerability and related risk - but definitely evaluate for the actual conditions:;

Example 1, a vulnerability that allows dumping all memory continuously: If the EC2 instances store any sort of sensitive information in memory as they deploy nodes (API keys, encryption keys, etc.) then this vulnerability might be a huge problem.

Example 2, a vulnerability that gives a shell: If the EC2 instance is configured in a way that users can't even run shells so compromising a specific piece of software that would normally grant a shell as the user won't work, then this vulnerability might be irrelevant.

Example 3, a vulnerability that allows the EC2 instance to be turned off: This might have no likely impact because another instance will just take over and the instance will be restarted.

Example 4, a vulnerability that allows writing to the root filesystem: irrelevant if the root filesystem is only mounted read only.

Example 5, a vulnerability that would allow them to deploy 5000 instances: probably a big deal?

I think a good situation to consider is the recent Binance hack, where the initial compromise was a system that had credentials that provided some control over the deployment pipeline. Solar Winds was kinda the same if I recall correctly.

Potential Agency Impact: If a common AMI is used to deploy independently operated, tenant-isolated environments for multiple federal agencies, should its potential agency impact be evaluated against the cumulative consequences across those agencies? Or should potential agency impact be determined from the realistic exploitation scenarios affecting the individual deployed environments?

As above, I think you need to consider both vectors. The most likely largest potential impact would be compromising the main image and the resulting command and control opportunities within other environments. Vulnerabilities within deployed images it sounds like would be limited to single-tenant exposure, but if they're in multiple environments and are all compromisable in the same way then realistically everyone is it risk right?

Grouping and remediation: Under VER-EVA-GRV, can vulnerabilities in the source AMI and its deployed instances be managed as a consolidated vulnerability group, provided that differences in exploitability and agency impact are documented?

Yes, grouping is designed very much for scenarios like this. It would allow you to report summary information like 77/112 instances have been redeployed with an updated AMI so there are only 35 instances now affected etc. You can then resize the group and remove the remediated nodes as desired. Or you can manage groups of groups. Whatever makes the most sense for your security engineering team for coordinating and managing the response.

If the deployed instances are patched but the source AMI remains vulnerable, does the continued existence and use of the image prevent the grouped vulnerability from being considered remediated?

This scenario can't exist if they're truly immutable instances? But as long as the source AMI might be deployed without a patch then it's definitely not remediated. If the AMI is disabled and would never be deployed again then you're fine.

Our question concerns the unit of vulnerability evaluation and reporting for reusable machine images, rather than whether maintaining updated images is a desirable security practice.

Yep, but it circles back to the same issue - how those are used varies from architecture to architecture, so providing specific requirements for every architecture is unreasonable.

As always, it boils down to the same critical factor: Do the right thing to secure your services, explain how you are doing so and why it's appropriate for your environment... and you're good as long as you're not blowing smoke.