tenable_io: replace scan details endpoint and remap schema

[chemamartinez]

Proposed commit message

tenable_io/data_stream/scan: fix scan details endpoint and remap schema

The scan data stream's CEL program fetched per-scan details via
GET /was/v2/scans/{id} (Web Application Scanning API) [1] instead of
the correct GET /scans/{id} (Vulnerability Management API) [2].
Because VM scan IDs are not valid in the WAS namespace, every Phase 2
request returned 404. The CEL 404 branch emitted a want_more stub that
the ingest pipeline dropped, so no scan documents were ever produced.
The bug was introduced in #16222; it escaped CI because the system
test mock was hand-authored to return 200 for the WAS path.

This change fixes the endpoint and rebuilds all schema-dependent
artifacts around the standard VM scan-details response:

- CEL program: /was/v2/scans/{id} → /scans/{id}.
- Fields: replace the WAS scan_details group with the full VM schema —
  info (+ acls[]), hosts, comphosts, vulnerabilities, compliance,
  history, notes, remediations, progress.
- Ingest pipeline: replace WAS ISO8601 date processors with UNIX
  processors for info timestamps and foreach processors for history[]
  dates; drop the UI-only filters[] array.
- Mock service: update test routes to /scans/{id} with standard-schema
  bodies so system tests exercise the real endpoint shape.
- Pipeline test fixture and sample event: updated to standard schema.
- Docs: update Phase 2 API reference link to scans-details.

[1] https://developer.tenable.com/reference/was-v2-scans-details
[2] https://developer.tenable.com/reference/scans-details

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

• Closes [Tenable IO] Fix scan data stream endpoint for scan details #20340

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package tenable_io 👍(2) 💚(2) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
scan	10204.08	5154.64	-5049.44 (-49.48%)	💔

To see the full report comment with /test benchmark fullreport

[mohitjha-elastic]

Why are both fields defined as keyword? Based on their descriptions, they appear to represent dates, so they should be mapped as date fields, with appropriate date processors on them.

[mohitjha-elastic]

The changelog entry is marked as type: bugfix under 4.13.0 (minor), but this change removes the published WAS scan_details.* fields and replaces them with the VM schema. Since it introduces a breaking schema change, it should be classified as a breaking-change and released as a major version.

If it's truly a bugfix, it should be a patch release (e.g., 4.12.1). As it stands, 4.13.0 with type: bugfix is inconsistent.

[efd6]

This is not a breaking change since the previous code never worked. However, yes, this should bump patch.

[chemamartinez]

I just thought it is a change it worths a minor version bump because of the size of the change. If you both think it should be a patch version despite the changes it includes that's ok for me.

[efd6]

The size of the change is not relevant to the version change. This is documented in the fleet wiki here.

[mohitjha-elastic]

VM schema adds hosts[].hostname and info.targets that often hold IPs. Other streams in this CDR package append to related.ip; can we also them to realted.ip?

[efd6]

… or related.hosts if it is not an IP.

[mohitjha-elastic]

The comphosts object is missing descriptions for its leaf fields. To keep the schema consistent, could we mirror the field descriptions from hosts (if not available for comphosts) onto the corresponding comphosts leaf fields?

[efd6]

Yes, that can be done. The schema is here hidden in a Markdown document.

[efd6]

… or related.hosts if it is not an IP.

[efd6]

Yes, that can be done. The schema is here hidden in a Markdown document.

[ShourieG]

@vera-review-bot review

[vera-review-bot]

Severity: 🟠 High confidence: high path: packages/tenable_io/data_stream/scan/elasticsearch/ingest_pipeline/default.yml:136

The new script splits scan_details.info.targets on commas and appends every token to related.ip without checking it is an IP literal, so CIDR ranges and FQDN targets land in an ECS ip field. Validate each token with the same IP regex already used for hosts[].hostname and route non-IP targets to related.hosts.

Details

related.ip is an ECS field of type ip, so it only accepts bare IPv4/IPv6 literals. Tenable's info.targets is a free-form, comma-delimited target specification that routinely holds CIDR ranges, IP ranges and FQDNs, not just single addresses. This PR's own mock data proves it: _dev/deploy/docker/files/config.yml returns "targets":"192.0.2.0/24" for scan 423, and that value is fed straight into ctx.related.ip. Because the logs-* index settings enable ignore_malformed, the document is not rejected — the value is silently discarded into _ignored and related.ip ends up empty for that scan, so the enrichment this PR adds does not work for the common case. The loop over details.hosts immediately below already guards with ipPat before deciding between related.ip and related.hosts; the targets loop is missing the same guard.

Recommendation:

Hoist the existing IP pattern and apply it to the target tokens as well, sending FQDN targets to related.hosts and skipping range/CIDR specifications:

  - script:
      description: Populate related.ip and related.hosts from scan details.
      lang: painless
      tag: populate_related_from_scan_details
      if: ctx.tenable_io?.scan?.scan_details != null
      source: |
        def details = ctx.tenable_io.scan.scan_details;
        def ipPat = /^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/;
        if (ctx.related == null) { ctx.related = new HashMap(); }
        if (details.info?.targets instanceof String) {
          for (def t : details.info.targets.splitOnToken(',')) {
            def target = t.trim();
            if (target.length() == 0) { continue; }
            if (ipPat.matcher(target).matches()) {
              if (ctx.related.ip == null) { ctx.related.ip = new ArrayList(); }
              if (!ctx.related.ip.contains(target)) { ctx.related.ip.add(target); }
            } else if (target.indexOf('/') < 0 && target.indexOf('-') < 0) {
              if (ctx.related.hosts == null) { ctx.related.hosts = new ArrayList(); }
              if (!ctx.related.hosts.contains(target)) { ctx.related.hosts.add(target); }
            }
          }
        }
        if (details.hosts instanceof List) {
          for (def host : (List) details.hosts) {
            if (host?.hostname != null && host.hostname.length() > 0) {
              if (ipPat.matcher(host.hostname).matches()) {
                if (ctx.related.ip == null) { ctx.related.ip = new ArrayList(); }
                if (!ctx.related.ip.contains(host.hostname)) { ctx.related.ip.add(host.hostname); }
              } else {
                if (ctx.related.hosts == null) { ctx.related.hosts = new ArrayList(); }
                if (!ctx.related.hosts.contains(host.hostname)) { ctx.related.hosts.add(host.hostname); }
              }
            }
          }
        }

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[efd6]

Do we know that we ever see IP ranges specified by dash? If not the suggested script will incorrectly discard target hostnames containing dashes without benefit.

[chemamartinez]

The script now validates each token from info.targets before routing it: bare IPv4 addresses go to related.ip, FQDNs and other non-IP strings go to related.hosts. Previously all tokens were blindly added to related.ip regardless of their format.

[efd6]

Tenable IO can return IPv6 addresses AFAICS.

[chemamartinez]

Added IPv6 addresses.

[vera-review-bot]

Severity: 🟡 Medium confidence: high path: packages/tenable_io/data_stream/scan/_dev/test/pipeline/test-scan.log:3

The only fixture event carrying scan_details uses a single bare IPv4 for both info.targets and hosts[].hostname, so neither the related.hosts branch nor non-literal targets are exercised. Add a fixture event with a CIDR/FQDN target list and a non-IP hostname.

Details

This PR introduces a new script processor with four distinct output paths (target -> related.ip, target -> non-IP, host -> related.ip, host -> related.hosts), but the fixture only sets "targets":"192.0.2.57" and "hostname":"192.0.2.57". Both take the same single branch, so related.hosts is never produced by the pipeline tests and the malformed-related.ip defect in finding 1 is invisible to CI even though the docker mock for scan 423 already returns a CIDR target. The scan_details.history date-conversion foreach is likewise only covered for a single-element array.

Recommendation:

Add a fourth fixture line whose scan_details exercises the remaining branches, and regenerate test-scan.log-expected.json:

{"control":true,"creation_date":1683043551,"enabled":true,"id":423,"last_modification_date":1683049400,"name":"Client Vulnerabiltiy Scan Group B","owner":"jdoe@​contoso.com","status":"completed","type":"remote","uuid":"a2389003-fec1-a45d-a45d-aece258c4133","scan_details":{"info":{"name":"Client Vulnerabiltiy Scan Group B","object_id":423,"status":"completed","scan_type":"remote","targets":"192.0.2.0/24,host1.example.com,192.0.2.57","scan_start":1683043551,"scan_end":1683049400,"timestamp":1683049400,"is_archived":false},"history":[{"history_id":1000423,"creation_date":1683043551,"last_modification_date":1683049400,"status":"completed","type":"remote"}],"hosts":[{"asset_id":3,"host_id":3,"hostname":"host1.example.com","host_index":0}]}}

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[vera-review-bot]

Severity: 🟠 High confidence: high path: packages/tenable_io/changelog.yml:5

This PR removes and restructures existing tenable_io.scan.scan_details.* field definitions, which is a breaking change, but ships it as a patch bump (4.12.0 -> 4.12.1) with type: bugfix; it should be a major bump (5.0.0) with type: breaking-change.

Details

The scan data stream's scan_details object is completely restructured by this PR. data_stream/scan/fields/fields.yml deletes 15 existing field declarations (scan_details.scan_id, user_id, config_id, target, created_at, updated_at, requested_action, status, and the whole metadata group with its 6 children) and replaces them with a different info/hosts/history/vulnerabilities tree. The ingest pipeline's date processors were repointed from scan_details.created_at/updated_at to scan_details.info.scan_start/scan_end/timestamp, and the CEL program now calls a different endpoint that returns a different payload shape.

Per the package versioning rules, a major bump is required for "field type changes or removals on existing integrations, ... data stream restructuring, default behavior changes that alter collected or normalized data". All three apply here. Shipping this as 4.12.1/bugfix means Fleet will auto-upgrade existing policies into a schema where every saved search, dashboard panel, detection rule or alert referencing tenable_io.scan.scan_details.status, .target, .scan_id or .metadata.* silently stops matching new documents, with no breaking-change notice in the changelog and no guard for the package upgrade test.

Recommendation:

Bump to a major version and mark the entry as a breaking change.

# packages/tenable_io/changelog.yml
# newer versions go on top
- version: "5.0.0"
  changes:
    - description: Replace the scan details endpoint with GET /scans/{id} and remap the scan_details fields to the standard VM scan-details schema.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/20347

and match it in the manifest:

# packages/tenable_io/manifest.yml
version: "5.0.0"

Also add a note to the existing ### Breaking Changes section of _dev/build/docs/README.md listing the removed scan_details fields and their replacements, so users can migrate saved objects.

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[vera-review-bot]

Severity: 🟡 Medium confidence: medium path: packages/tenable_io/data_stream/scan/elasticsearch/ingest_pipeline/default.yml:139

The new populate_related_from_scan_details script only recognises dotted-quad IPv4 as an IP, so IPv6 addresses and IP ranges from scan_details.info.targets land in related.hosts instead of related.ip, and CIDR targets are dropped entirely; classify with a grok/convert-style IP check or at minimum handle ranges and IPv6.

Details

scan_details.info.targets is a comma-delimited target list that Tenable populates with IPv4 addresses, IPv4 ranges (192.0.2.1-192.0.2.20), CIDR subnets, hostnames, FQDNs and IPv6 addresses. The script classifies each token with a single rule: skip anything containing /, send anything matching ^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$ to related.ip, and send everything else to related.hosts.

As written that means (a) a CIDR target such as the 192.0.2.0/24 in _dev/test/pipeline/test-scan.log is discarded and never surfaces in related.* at all, and (b) an IPv6 target or an IPv4 range is appended to related.hosts, which is defined by ECS as "all hostnames or other host identifiers" - so IP data ends up in the hostname field and analysts pivoting on related.ip miss those scans. The same misclassification applies to the scan_details.hosts[].hostname loop below, since Tenable populates hostname with the IP when no name resolves. The related.hosts field is a keyword so nothing fails at index time; the data is just wrong.

Recommendation:

Recognise the range/CIDR/IPv6 forms explicitly instead of falling through to related.hosts. For example, normalise a range or subnet to its base address and use a broader IP test:

  - script:
      description: Populate related.ip and related.hosts from scan details.
      lang: painless
      tag: populate_related_from_scan_details
      if: ctx.tenable_io?.scan?.scan_details != null
      source: |
        def details = ctx.tenable_io.scan.scan_details;
        def ipv4Pat = /^(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}$/;
        def ipv6Pat = /^[0-9A-Fa-f:]*:[0-9A-Fa-f:.]*$/;
        if (ctx.related == null) { ctx.related = new HashMap(); }
        def addIp(Map related, String v) {
          if (related.ip == null) { related.ip = new ArrayList(); }
          if (!related.ip.contains(v)) { related.ip.add(v); }
        }
        def addHost(Map related, String v) {
          if (related.hosts == null) { related.hosts = new ArrayList(); }
          if (!related.hosts.contains(v)) { related.hosts.add(v); }
        }
        def classify(Map related, String raw) {
          def v = raw.trim();
          if (v.length() == 0) { return; }
          // 192.0.2.0/24 -> 192.0.2.0 ; 192.0.2.1-192.0.2.20 -> 192.0.2.1
          def base = v;
          int slash = base.indexOf('/');
          if (slash >= 0) { base = base.substring(0, slash); }
          int dash = base.indexOf('-');
          if (dash > 0 && base.indexOf(':') < 0) { base = base.substring(0, dash); }
          base = base.trim();
          if (ipv4Pat.matcher(base).matches() || ipv6Pat.matcher(base).matches()) {
            addIp(related, base);
          } else {
            addHost(related, v);
          }
        }
        if (details.info?.targets instanceof String) {
          for (def t : details.info.targets.splitOnToken(',')) { classify(ctx.related, t); }
        }
        if (details.hosts instanceof List) {
          for (def host : (List) details.hosts) {
            if (host?.hostname instanceof String && host.hostname.length() > 0) {
              classify(ctx.related, host.hostname);
            }
          }
        }

Then extend _dev/test/pipeline/test-scan.log with a target list containing a range and an IPv6 address so the new branches are covered.

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[vera-review-bot]

Severity: 🟡 Medium confidence: low path: packages/tenable_io/data_stream/scan/elasticsearch/ingest_pipeline/default.yml:107

scan_details.info.scanner_start/scanner_end are parsed as ISO8601 while their siblings scan_start, scan_end and timestamp in the same info object are parsed as UNIX; if they are epoch values the parse silently fails and the raw number is indexed into a date field as epoch millis.

Details

Every other timestamp this PR adds from the /scans/{id} info object (scan_start, scan_end, timestamp) and from history[] (creation_date, last_modification_date) is parsed with formats: [UNIX]. scanner_start and scanner_end, which come from the same info object and describe the same scan clock, are instead parsed with formats: [ISO8601] plus ignore_failure: true.

Because of ignore_failure: true a format mismatch is silent: the original value stays in place and is indexed into tenable_io.scan.scan_details.info.scanner_start, which fields.yml declares as type: date. Elasticsearch's default date parser treats a bare number as epoch milliseconds, so an epoch-seconds value such as 1683282785 would be stored as 1970-01-20 rather than 2023-05-05, with no error surfaced anywhere.

Neither branch is exercised by the tests: both _dev/test/pipeline/test-scan.log fixtures and both _dev/deploy/docker/files/config.yml mocks set scanner_start/scanner_end to null, so the ISO8601 choice is untested. Please confirm the wire format against a real /scans/{id} response before merge.

Recommendation:

If these fields are epoch values like their siblings, parse them the same way and cover them in the fixture:

  - date:
      field: json.scan_details.info.scanner_start
      target_field: json.scan_details.info.scanner_start
      tag: date_scan_details_scanner_start
      if: ctx.json?.scan_details?.info?.scanner_start != null && ctx.json.scan_details.info.scanner_start != ''
      formats:
        - UNIX
        - ISO8601
      ignore_failure: true
  - date:
      field: json.scan_details.info.scanner_end
      target_field: json.scan_details.info.scanner_end
      tag: date_scan_details_scanner_end
      if: ctx.json?.scan_details?.info?.scanner_end != null && ctx.json.scan_details.info.scanner_end != ''
      formats:
        - UNIX
        - ISO8601
      ignore_failure: true

Listing both formats keeps imported-scan payloads working either way, and adding a fixture line with non-null scanner_start/scanner_end makes the behaviour verifiable.

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[vera-review-bot]

Severity: 🔵 Low confidence: high path: packages/tenable_io/data_stream/scan/elasticsearch/ingest_pipeline/default.yml:69

None of the nine processors added in this change carry a tag, so pipeline failures on the new scan_details date parsing cannot be attributed to a specific processor; add tag (and a short description) to each.

Details

The three date processors for scan_details.info.scan_start/scan_end/timestamp, the two foreach wrappers over scan_details.history, the two date processors for scanner_start/scanner_end, and the remove for scan_details.filters are all added without a tag. The pipeline-level on_failure appends {{{ _ingest.on_failure_message }}} to error.message, but without a processor tag the resulting pipeline_error document does not identify which of the eight new date conversions failed, which makes triaging a malformed scan_details payload guesswork. The script processor added in the same change does set tag: populate_related_from_scan_details, so the pipeline is inconsistent with itself.

Processor tags become enforced by elastic-package check at format_version >= 3.6.0; this package is on 3.4.0, so this is an improvement rather than a lint failure today.

Recommendation:

Add a tag to each new processor, for example:

  - date:
      field: json.scan_details.info.scan_start
      target_field: json.scan_details.info.scan_start
      tag: date_scan_details_info_scan_start
      if: ctx.json?.scan_details?.info?.scan_start != null && ctx.json.scan_details.info.scan_start != ''
      formats:
        - UNIX
  - foreach:
      field: json.scan_details.history
      tag: foreach_scan_details_history_creation_date
      ignore_missing: true
      processor:
        date:
          field: _ingest._value.creation_date
          target_field: _ingest._value.creation_date
          tag: date_scan_details_history_creation_date
          formats:
            - UNIX
          ignore_failure: true
  - remove:
      field: json.scan_details.filters
      tag: remove_scan_details_filters
      description: Scan filter definitions are configuration noise and are not mapped.
      ignore_missing: true

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[vera-review-bot]

Review summary

Issues found across the latest commits 2b1de59 — 1 high, 2 medium, 1 low

• 🟠 This PR removes and restructures existing tenable_io.scan.scan_details.* field definitions, which is a breaking change, but ships it as a patch bump (4.12.0 -> 4.12.1) with type: bugfix (link) (Unresolved)
• 🟡 The new populate_related_from_scan_details script only recognises dotted-quad IPv4 as an IP, so IPv6 addresses and IP ranges from scan_details.info.targets land in related.hosts instead of related.ip, and CIDR targets are dropped entirely (link) (Unresolved)
• 🟡 scan_details.info.scanner_start/scanner_end are parsed as ISO8601 while their siblings scan_start, scan_end and timestamp in the same info object are parsed as UNIX (link) (Unresolved)
• 🔵 None of the nine processors added in this change carry a tag, so pipeline failures on the new scan_details date parsing cannot be attributed to a specific processor (link) (Unresolved)

Issues found across earlier commits 64ff440 — 1 high, 1 medium

• 🟠 The new script splits scan_details.info.targets on commas and appends every token to related.ip without checking it is an IP literal, so CIDR ranges and FQDN targets land in an ECS ip field. Validate each token with the same IP regex already used for hosts[].hostname and route non-IP targets to related.hosts. (link) (Unresolved)
• 🟡 The only fixture event carrying scan_details uses a single bare IPv4 for both info.targets and hosts[].hostname, so neither the related.hosts branch nor non-literal targets are exercised. Add a fixture event with a CIDR/FQDN target list and a non-IP hostname. (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[mohitjha-elastic]

LGTM for my comments, please wait for @efd6 !!

[mohitjha-elastic]

Fields are now date, but processors use ISO8601, while other sibling date fields use UNIX. Tests still have null, so the path is untested. Please verify it once if missed.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 2218b9f

History

• 💚 Build #46771 succeeded 222304d
• 💚 Build #46735 succeeded 2b1de59
• 💚 Build #46684 succeeded 64ff440
• 💚 Build #46596 succeeded 7f25523

cc @chemamartinez

[elastic-vault-github-plugin-prod]

Package tenable_io - 4.12.1 containing this change is available at https://epr.elastic.co/package/tenable_io/4.12.1/