aws.inspector: fix CLOSED/SUPPRESSED findings staying ACTIVE by kcreddy · Pull Request #19918 · elastic/integrations

kcreddy · 2026-07-01T18:42:21Z

Proposed commit message

aws: fix Inspector CLOSED/SUPPRESSED findings staying ACTIVE in Elastic

AWS Inspector findings that were marked CLOSED or SUPPRESSED in AWS
remained ACTIVE in Elastic, inflating active vulnerability counts on
the Vulnerability Findings page. Two independent defects caused this.

Ingestion gap: the httpjson input polled and advanced its cursor on
lastObservedAt, which AWS freezes once it stops observing a finding.
Status transitions to CLOSED/SUPPRESSED (which move updatedAt, not
lastObservedAt) fell outside the poll window and were never collected.
Switch the filter to updatedAt. Because the Inspector API cannot sort
by updatedAt, the cursor stores the poll time as a time-based
high-water mark instead of the last event's timestamp.

Transform gap: the latest transform source query filtered on
aws.inspector.status: ACTIVE, so SUPPRESSED/CLOSED documents never
entered the transform and could not displace the stale ACTIVE record.
Remove that filter so the latest document per finding reflects the
current status, and bump fleet_transform_version to reinstall.

Map aws.inspector.status to the ECS vulnerability.status field
(ACTIVE/SUPPRESSED -> open, CLOSED -> fixed) so remediated findings
can be filtered out on the Vulnerability Findings page. SUPPRESSED is
mapped to open because the vulnerability is still present; users can
hide it with NOT aws.inspector.status: SUPPRESSED. Document this in
the Inspector README and add the vulnerability.status field to the
data stream and transform mappings.

Add a second mock finding (a CLOSED vulnerability) to the Inspector 
system-test config so status transitions are exercised end to end. 
Use relative epoch timestamps with updatedAt later than the 
observation times to reflect the finding being closed after it was last
observed.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Screenshots

Confirmed transform index now populates CLOSED vulnerabilities.

AWS Inspector findings that were marked CLOSED or SUPPRESSED in AWS remained ACTIVE in Elastic, inflating active vulnerability counts on the Vulnerability Findings page. Two independent defects caused this. Ingestion gap: the httpjson input polled and advanced its cursor on lastObservedAt, which AWS freezes once it stops observing a finding. Status transitions to CLOSED/SUPPRESSED (which move updatedAt, not lastObservedAt) fell outside the poll window and were never collected. Switch the filter to updatedAt. Because the Inspector API cannot sort by updatedAt, the cursor stores the poll time as a time-based high-water mark instead of the last event's timestamp. Transform gap: the latest transform source query filtered on aws.inspector.status: ACTIVE, so SUPPRESSED/CLOSED documents never entered the transform and could not displace the stale ACTIVE record. Remove that filter so the latest document per finding reflects the current status, and bump fleet_transform_version to reinstall. Map aws.inspector.status to the ECS vulnerability.status field (ACTIVE/SUPPRESSED -> open, CLOSED -> fixed) so remediated findings can be filtered out on the Vulnerability Findings page. SUPPRESSED is mapped to open because the vulnerability is still present; users can hide it with NOT aws.inspector.status: SUPPRESSED. Document this in the Inspector README and add the vulnerability.status field to the data stream and transform mappings.

github-actions · 2026-07-01T18:44:14Z

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

…fsets the poll cursor to tolerate delay between when a finding's `updatedAt` is set and when it becomes available from the API, preventing missed findings.

infra-vault-gh-plugin-prod · 2026-07-03T08:09:58Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

ShourieG

LGTM

vera-review-bot · 2026-07-03T09:31:30Z


+## Vulnerability status
+
+The integration maps the Amazon Inspector finding status (`aws.inspector.status`) to the ECS `vulnerability.status` field so the [Vulnerability Findings page](https://www.elastic.co/docs/solutions/security/cloud/findings-page-3) can distinguish active from remediated findings:


Severity: 🔵 Low confidence: high path: packages/aws/_dev/build/docs/inspector.md:54

Docs call vulnerability.status an ECS field, but it is a custom package field; reword to avoid implying ECS provenance.

Details

The new section states the integration maps aws.inspector.status to "the ECS vulnerability.status field". ECS has no vulnerability.status field (the ECS vulnerability field set ends at vulnerability.severity). This PR correctly defines vulnerability.status as a custom keyword in fields/vulnerability.yml, so it is a package-defined field, not ECS. The same inaccurate "ECS" phrasing also appears in changelog.yml (6.21.0 enhancement entry) and in the ingest_pipeline/default.yml comment ("ECS vulnerability lifecycle field"). Calling a custom field an ECS field can mislead maintainers about its provenance and mapping guarantees.

Recommendation:

Describe it as a custom field mapped alongside ECS vulnerability.* fields rather than an ECS field. For example:

The integration maps the Amazon Inspector finding status (`aws.inspector.status`) to the package field `vulnerability.status` so the [Vulnerability Findings page](https://www.elastic.co/docs/solutions/security/cloud/findings-page-3) can distinguish active from remediated findings:

Apply the same wording fix to the changelog entry and the pipeline comment for consistency.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@kcreddy - seems right according to https://www.elastic.co/docs/reference/ecs/ecs-vulnerability

Yeah the field is still in RFC until 9.5. Changed the wording in 2ce4301.

I am more concerned about the bot finding these pre-existing issues late. The bot's earlier review had no issues: #19918 (comment) and my latest commit had nothing to do with the current issue.
I found this has been case with community PRs as well. The community author fixes the bot's current issues, and then the bot finds new issues (not related to latest commit, but from much earlier commits).
I think it causes slower merges, increased CI costs, and also reviewer fatigue.

vera-review-bot · 2026-07-03T09:31:32Z

Review summary

Issues found across the latest commits 5aefd1f — 1 low

🔵 Docs call vulnerability.status an ECS field, but it is a custom package field (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

github-actions · 2026-07-03T09:32:21Z

TL;DR

Check integrations aws failed because packages/aws/docs/inspector.md is stale. The committed generated README example still has the older Inspector sample event timestamp (2026-06-03T08:01:42Z), while packages/aws/data_stream/inspector/sample_event.json at this commit has the newer generated event (2026-06-03T09:09:10Z).

Remediation

Rebuild the AWS package docs from the PR branch with elastic-package build from packages/aws, then commit the updated packages/aws/docs/inspector.md generated from the current sample_event.json.
Re-run .buildkite/scripts/test_one_package.sh packages/aws origin/main 5aefd1f1adab787358a6f344a35f012362eb22b1 or the Check integrations aws Buildkite step.

Investigation details

Root Cause

This is generated documentation drift. Buildkite regenerated inspector.md and compared it with the committed file; the generated output uses the current packages/aws/data_stream/inspector/sample_event.json, but the committed packages/aws/docs/inspector.md still contains the previous sample render.

Evidence in the PR head:

packages/aws/docs/inspector.md:76
    "`@timestamp`": "2026-06-03T08:01:42.000Z",

packages/aws/docs/inspector.md:103
    "last_observed_at": "2026-06-03T08:01:42.000Z",

packages/aws/data_stream/inspector/sample_event.json:2
    "`@timestamp`": "2026-06-03T09:09:10.000Z",

packages/aws/data_stream/inspector/sample_event.json:29
    "last_observed_at": "2026-06-03T09:09:10.000Z",

The sample event timestamps come from the Inspector mock response template, which uses relative now values:

packages/aws/data_stream/inspector/_dev/deploy/docker/files/config.yml:34
    "lastObservedAt": {{ (now "-720h").Unix }},

packages/aws/data_stream/inspector/_dev/deploy/docker/files/config.yml:182
    "updatedAt": {{ (now "-720h").Unix }}

So after regenerating sample_event.json, the package docs also need to be regenerated and committed.

Evidence

Build: https://buildkite.com/elastic/integrations/builds/45457
Job/step: Check integrations aws
Key log excerpt:

inspector.md is outdated. Rebuild the package with 'elastic-package build'

@@ -75,7 +75,7 @@
-    "`@timestamp`": "2026-06-03T08:01:42.000Z",
+    "`@timestamp`": "2026-06-03T09:09:10.000Z",
...
-            "last_observed_at": "2026-06-03T08:01:42.000Z",
+            "last_observed_at": "2026-06-03T09:09:10.000Z",

Verification

Not run locally; this workflow is read-only. The conclusion is based on the Buildkite log plus the PR head contents for sample_event.json, docs/inspector.md, and the Inspector mock config.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

elastic-vault-github-plugin-prod · 2026-07-03T10:22:46Z

✅ All changelog entries have the correct PR link.

infra-vault-gh-plugin-prod · 2026-07-03T10:30:04Z

💔 Build Failed

Buildkite Build
Commit: 59d72f6

Failed CI Steps

Check integrations aws

History

💔 Build #45460 failed 2ce4301
💔 Build #45457 failed 5aefd1f
💔 Build #45451 failed 532f844
💔 Build #45403 failed 1dd0af3
💔 Build #45343 failed 7243506

cc @kcreddy

kcreddy added 2 commits July 2, 2026 00:11

update pr number

7243506

andrewkroh added Integration:aws AWS documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. labels Jul 1, 2026

Add a lag_time variable to the Amazon Inspector data stream that of…

1dd0af3

…fsets the poll cursor to tolerate delay between when a finding's `updatedAt` is set and when it becomes available from the API, preventing missed findings.

This comment has been minimized.

Sign in to view

Add test fixture for CLOSED vuln

532f844

kcreddy added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Jul 3, 2026

kcreddy self-assigned this Jul 3, 2026

kcreddy marked this pull request as ready for review July 3, 2026 08:09

kcreddy requested review from a team as code owners July 3, 2026 08:09

kcreddy added enhancement New feature or request bugfix Pull request that fixes a bug issue labels Jul 3, 2026

ShourieG reviewed Jul 3, 2026

View reviewed changes

Comment thread packages/aws/changelog.yml

kcreddy requested a review from ShourieG July 3, 2026 08:45

ShourieG reviewed Jul 3, 2026

View reviewed changes

Comment thread packages/aws/data_stream/inspector/elasticsearch/ingest_pipeline/default.yml

ShourieG reviewed Jul 3, 2026

View reviewed changes

Comment thread packages/aws/data_stream/inspector/agent/stream/httpjson.yml.hbs

Address review comments

5aefd1f

kcreddy requested a review from ShourieG July 3, 2026 09:10

ShourieG approved these changes Jul 3, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

vera-review-bot Bot reviewed Jul 3, 2026

View reviewed changes

Address review comments

2ce4301

Fix README

59d72f6


		## Vulnerability status

		The integration maps the Amazon Inspector finding status (`aws.inspector.status`) to the ECS `vulnerability.status` field so the [Vulnerability Findings page](https://www.elastic.co/docs/solutions/security/cloud/findings-page-3) can distinguish active from remediated findings:

Uh oh!

Conversation

kcreddy commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Screenshots

Uh oh!

github-actions Bot commented Jul 1, 2026

✅ Elastic Docs Style Checker (Vale)

Uh oh!

This comment has been minimized.

This comment has been minimized.

infra-vault-gh-plugin-prod Bot commented Jul 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ShourieG left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

vera-review-bot Bot Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

ShourieG Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

kcreddy Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vera-review-bot Bot commented Jul 3, 2026

Review summary

Uh oh!

github-actions Bot commented Jul 3, 2026

TL;DR

Remediation

Root Cause

Evidence

Verification

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 3, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kcreddy commented Jul 1, 2026 •

edited

Loading

kcreddy Jul 3, 2026 •

edited

Loading

infra-vault-gh-plugin-prod Bot commented Jul 3, 2026 •

edited

Loading