Skill Being Reviewed
Skill name: aws-review
Skill path: skills/cloud/aws-review/
False Positive Analysis
Benign configuration that can trigger an overconfident failure/pass:
resource "aws_cloudtrail" "org" {
is_multi_region_trail = true
is_organization_trail = true
}Why this can be misleading:
The skill evaluates AWS controls from IaC, CLI output, or config exports, but it does not require an evidence-confidence field. An organization trail in a delegated security account can satisfy broad logging intent, while account-level or region-level gaps may be impossible to prove from one Terraform module alone. The result should be "Pass with organization-scope evidence," "Fail," or "Not Evaluable," not a bare pass/fail.
Coverage Gaps
Missed variant 1: Account-scope and organization-scope evidence are not separated
Evidence source: Terraform module for security account only.
Control: CloudTrail enabled in all accounts and all regions.
Why it should be caught:
AWS Organizations and CloudTrail delegated administration can centralize controls, but an IaC-only review may not show every member account, region, delegated admin, and service-linked-role prerequisite. The skill should record scope coverage and evidence confidence.
Missed variant 2: Access Analyzer policy validation is conflated with analyzer enablement
CIS 1.20: aws_accessanalyzer_analyzer exists.
IAM policy review: no policy validation findings or analyzer coverage evidence.
Why it should be caught:
Enabling Access Analyzer is different from using policy validation findings while reviewing IAM policy JSON. The skill should distinguish analyzer deployment evidence from policy-validation evidence. Paid custom policy checks should remain optional and should not be required for this no-upfront workflow.
Missed variant 3: "Not Evaluable" controls lack reason codes
CIS 1.1 contact details: not visible in Terraform.
Result: Not Evaluable.
Reason: live-account-only control; no CLI/export evidence supplied.
Why it should be caught:
The current output has a Not Evaluable count, but detailed findings do not capture why evidence was insufficient. Without a reason code, maintainers cannot distinguish live-account-only controls, missing regions, missing member-account coverage, or unsupported IaC provider coverage.
Edge Cases
- Organization trails can be valid, but the reviewer must still document management/delegated admin scope, all-region intent, S3/KMS policy evidence, and CloudWatch/logging integration evidence.
- AWS Config, Access Analyzer, Security Hub, and CloudTrail are regional services or have region-sensitive evidence; a single resource in one region should not imply all-region coverage.
- IAM Access Analyzer custom policy checks can incur charges; the skill should not make those mandatory.
- Contact details and root-account evidence may require live exports; IaC absence alone should be "Not Evaluable" unless there is clear contradictory evidence.
Remediation Quality
Comparison to Other Tools
| Tool |
Catches this? |
Notes |
| AWS Security Hub |
Partial |
Can evaluate many controls live, but not all IaC intent or reason codes. |
| AWS Config |
Partial |
Gives live regional configuration signals, but coverage itself must be verified. |
| IAM Access Analyzer |
Partial |
Analyzer deployment and policy validation are separate evidence streams. |
Overall Assessment
Strengths:
- Clear CIS AWS Foundations v3.0.0 structure.
- Useful severity examples and common pitfalls.
- Good pointer to the detailed benchmark checklist.
Needs improvement:
- Add evidence-confidence levels: IaC-only, live export, organization-wide, sampled, or unknown.
- Add multi-account/all-region scope fields for regional and organization controls.
- Add Not Evaluable reason codes.
- Clarify Access Analyzer enablement versus policy validation evidence, with paid custom checks optional only.
Priority recommendations:
- Add a scope/evidence-quality review step before final reporting.
- Expand output with evidence source, scope coverage, and Not Evaluable reason.
- Add pitfalls for Access Analyzer policy validation and organization/delegated-admin evidence.
References used:
Bounty Info
Skill Being Reviewed
Skill name:
aws-reviewSkill path:
skills/cloud/aws-review/False Positive Analysis
Benign configuration that can trigger an overconfident failure/pass:
Why this can be misleading:
The skill evaluates AWS controls from IaC, CLI output, or config exports, but it does not require an evidence-confidence field. An organization trail in a delegated security account can satisfy broad logging intent, while account-level or region-level gaps may be impossible to prove from one Terraform module alone. The result should be "Pass with organization-scope evidence," "Fail," or "Not Evaluable," not a bare pass/fail.
Coverage Gaps
Missed variant 1: Account-scope and organization-scope evidence are not separated
Why it should be caught:
AWS Organizations and CloudTrail delegated administration can centralize controls, but an IaC-only review may not show every member account, region, delegated admin, and service-linked-role prerequisite. The skill should record scope coverage and evidence confidence.
Missed variant 2: Access Analyzer policy validation is conflated with analyzer enablement
Why it should be caught:
Enabling Access Analyzer is different from using policy validation findings while reviewing IAM policy JSON. The skill should distinguish analyzer deployment evidence from policy-validation evidence. Paid custom policy checks should remain optional and should not be required for this no-upfront workflow.
Missed variant 3: "Not Evaluable" controls lack reason codes
Why it should be caught:
The current output has a Not Evaluable count, but detailed findings do not capture why evidence was insufficient. Without a reason code, maintainers cannot distinguish live-account-only controls, missing regions, missing member-account coverage, or unsupported IaC provider coverage.
Edge Cases
Remediation Quality
Comparison to Other Tools
Overall Assessment
Strengths:
Needs improvement:
Priority recommendations:
References used:
Bounty Info
1005150221@qq.com