crowdstrike: rename FDR lookup destinations out of logs-* namespace

packages/crowdstrike/_dev/build/docs/README.md

@@ -314,23 +314,36 @@ The integration sets `event.severity` according to the mapping in the table abov
 | 60 - 79                | high          |
 | 80 - 100               | critical      |
 
+### Lookup index aliases renamed in 3.16.2
+
+In 3.16.2 the FDR lookup transform destination indices and stable aliases were moved out of the `logs-*` namespace so the empty lookup indices no longer match the default Security Solution `logs-*` index pattern (which produced "missing the timestamp field `@timestamp`" warnings on detection rules):
+
+| Before                                            | After                                       |
+|---------------------------------------------------|---------------------------------------------|
+| `logs-crowdstrike_lookup.aidmaster`               | `crowdstrike_lookup.aidmaster`              |
+| `logs-crowdstrike_lookup.userinfo`                | `crowdstrike_lookup.userinfo`               |
+| `logs-crowdstrike_lookup.dest_aidmaster-1`        | `crowdstrike_lookup.dest_aidmaster-1`       |
+| `logs-crowdstrike_lookup.dest_userinfo-1`         | `crowdstrike_lookup.dest_userinfo-1`        |
+
+If you wrote custom ES|QL queries, dashboards, or detection rules against the old alias names, update them to the new names. The bundled dashboards have been updated. After upgrading, the old `logs-crowdstrike_lookup.*` indices and aliases left behind by previous installs are unused and can be safely deleted.
+
 ### Query-time host metadata enrichment (LOOKUP JOIN)
 
 When the integration is installed, a transform maintains the latest host metadata (`aidmaster`) per host in a lookup index. You can enrich FDR event data with this metadata at query time using ES|QL [`LOOKUP JOIN`](https://www.elastic.co/docs/reference/query-languages/esql/commands/lookup-join) on `host.id`.
 
-**Lookup index:** `logs-crowdstrike_lookup.aidmaster` — stable alias for the aidmaster lookup data maintained by the integration transform. The backing destination index is managed by the package and may change when you upgrade; use this alias in queries so joins keep working across versions. The lookup retains only `host.id` and `crowdstrike.info.*`; ECS host fields from `aidmaster` are stored under `crowdstrike.info.host.*` (e.g. `crowdstrike.info.host.hostname`, `crowdstrike.info.host.cid`, `crowdstrike.info.host.os_version`).
+**Lookup index:** `crowdstrike_lookup.aidmaster` — stable alias for the aidmaster lookup data maintained by the integration transform. The backing destination index is managed by the package and may change when you upgrade; use this alias in queries so joins keep working across versions. The lookup retains only `host.id` and `crowdstrike.info.*`; ECS host fields from `aidmaster` are stored under `crowdstrike.info.host.*` (e.g. `crowdstrike.info.host.hostname`, `crowdstrike.info.host.cid`, `crowdstrike.info.host.os_version`).
 
 **Example ES|QL query:**
 
 ```sql
 FROM logs-crowdstrike.fdr-*
 | WHERE aws.s3.object.key LIKE "*/data/*"
-| LOOKUP JOIN logs-crowdstrike_lookup.aidmaster ON host.id
+| LOOKUP JOIN crowdstrike_lookup.aidmaster ON host.id
 | KEEP @timestamp, event.action, host.id, crowdstrike.info.host.hostname
 | LIMIT 20
 ```
 
-**Elasticsearch 8.19+** is required for `LOOKUP JOIN` to resolve an alias. Use `logs-crowdstrike_lookup.aidmaster` as in the example above. On **releases before 8.19**, `LOOKUP JOIN` must target the concrete transform destination index instead: in Kibana go to **Stack Management** → **Transforms**, open the CrowdStrike latest aidmaster transform, and use the **destination_index** name shown there (that name can change with the integration version).
+**Elasticsearch 8.19+** is required for `LOOKUP JOIN` to resolve an alias. Use `crowdstrike_lookup.aidmaster` as in the example above. On **releases before 8.19**, `LOOKUP JOIN` must target the concrete transform destination index instead: in Kibana go to **Stack Management** → **Transforms**, open the CrowdStrike latest aidmaster transform, and use the **destination_index** name shown there (that name can change with the integration version).
 
 **Using enriched fields:** Enrichment from the lookup is under the `crowdstrike.info.host.*` namespace (e.g. `crowdstrike.info.host.hostname` for hostname, `crowdstrike.info.host.cid` for customer ID). Use these fields in dashboards and detection rules when building on query-time enrichment.
 
@@ -340,7 +353,7 @@ FROM logs-crowdstrike.fdr-*
 
 A second transform maintains the latest user metadata per host-user pair from `UserIdentity` and `UserLogon` sensor events in a lookup index. Unlike `userinfo` directory data (which requires [Falcon Discover](https://www.crowdstrike.com/platform/exposure-management/falcon-discover/) and covers only Windows), sensor events are available to all FDR customers on all platforms (Windows, macOS, Linux, ChromeOS). You can enrich FDR events with user metadata at query time using ES|QL [`LOOKUP JOIN`](https://www.elastic.co/docs/reference/query-languages/esql/commands/lookup-join).
 
-**Lookup index:** `logs-crowdstrike_lookup.userinfo` — stable alias for the user lookup data maintained by the integration transform. The backing destination index is managed by the package and may change when you upgrade; use this alias in queries so joins keep working across versions. The lookup retains only `host_user_key` and `crowdstrike.info.*`; user fields are stored under `crowdstrike.info.user.*` (e.g. `crowdstrike.info.user.name`, `crowdstrike.info.user.domain`, `crowdstrike.info.user.logon_type`).
+**Lookup index:** `crowdstrike_lookup.userinfo` — stable alias for the user lookup data maintained by the integration transform. The backing destination index is managed by the package and may change when you upgrade; use this alias in queries so joins keep working across versions. The lookup retains only `host_user_key` and `crowdstrike.info.*`; user fields are stored under `crowdstrike.info.user.*` (e.g. `crowdstrike.info.user.name`, `crowdstrike.info.user.domain`, `crowdstrike.info.user.logon_type`).
 
 **Composite join key:** Because Unix UIDs are local to each host (the same numeric UID can refer to different users on different machines), the user lookup uses a composite key combining both `host.id` and `user.id`. Queries must construct this key with `EVAL` before joining:
 
@@ -352,7 +365,7 @@ A second transform maintains the latest user metadata per host-user pair from `U
 FROM logs-crowdstrike.fdr-*
 | WHERE aws.s3.object.key LIKE "*/data/*" OR log.file.path LIKE "*/data/*"
 | EVAL host_user_key = CONCAT(host.id, "::", user.id)
-| LOOKUP JOIN logs-crowdstrike_lookup.userinfo ON host_user_key
+| LOOKUP JOIN crowdstrike_lookup.userinfo ON host_user_key
 | KEEP @timestamp, event.action, host.id, user.id,
        crowdstrike.info.user.name, crowdstrike.info.user.domain
 | LIMIT 20
@@ -363,15 +376,15 @@ FROM logs-crowdstrike.fdr-*
 ```sql
 FROM logs-crowdstrike.fdr-*
 | WHERE aws.s3.object.key LIKE "*/data/*" OR log.file.path LIKE "*/data/*"
-| LOOKUP JOIN logs-crowdstrike_lookup.aidmaster ON host.id
+| LOOKUP JOIN crowdstrike_lookup.aidmaster ON host.id
 | EVAL host_user_key = CONCAT(host.id, "::", user.id)
-| LOOKUP JOIN logs-crowdstrike_lookup.userinfo ON host_user_key
+| LOOKUP JOIN crowdstrike_lookup.userinfo ON host_user_key
 | KEEP @timestamp, event.action, host.id, crowdstrike.info.host.hostname,
        user.id, crowdstrike.info.user.name, crowdstrike.info.user.domain
 | LIMIT 20
 ```
 
-**Elasticsearch 8.19+** is required for `LOOKUP JOIN` to resolve an alias. Use `logs-crowdstrike_lookup.userinfo` as in the examples above. On **releases before 8.19**, `LOOKUP JOIN` must target the concrete transform destination index instead: in Kibana go to **Stack Management** → **Transforms**, open the CrowdStrike latest userinfo transform, and use the **destination_index** name shown there (that name can change with the integration version). If you use both host and user lookups on releases before 8.19, you will need two concrete destination index names — one for aidmaster and one for userinfo — both obtainable from **Stack Management** → **Transforms**.
+**Elasticsearch 8.19+** is required for `LOOKUP JOIN` to resolve an alias. Use `crowdstrike_lookup.userinfo` as in the examples above. On **releases before 8.19**, `LOOKUP JOIN` must target the concrete transform destination index instead: in Kibana go to **Stack Management** → **Transforms**, open the CrowdStrike latest userinfo transform, and use the **destination_index** name shown there (that name can change with the integration version). If you use both host and user lookups on releases before 8.19, you will need two concrete destination index names — one for aidmaster and one for userinfo — both obtainable from **Stack Management** → **Transforms**.
 
 **Using enriched fields:** Enrichment from the user lookup is under the `crowdstrike.info.user.*` namespace (e.g. `crowdstrike.info.user.name` for username, `crowdstrike.info.user.domain` for UPN domain, `crowdstrike.info.user.logon_type` for logon type). Use these fields in dashboards and ES|QL detection rules when building on query-time enrichment. Note that detection rules using EQL, threshold, or KQL operate on stored documents and cannot use `LOOKUP JOIN` — those rule types continue to rely on ingest-time cache enrichment for user metadata.
 

packages/crowdstrike/changelog.yml

@@ -1,4 +1,13 @@
 # newer versions go on top
+- version: "3.16.2"
+  changes:
+    - description: >-
+        Rename FDR lookup transform destinations and aliases from `logs-crowdstrike_lookup.*` to
+        `crowdstrike_lookup.*` so empty lookup indices no longer match the Security Solution `logs-*`
+        index pattern and trigger missing-`@timestamp` warnings. Old `logs-crowdstrike_lookup.*`
+        indices from previous installs can be safely deleted after upgrade.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/19005
 - version: "3.16.1"
   changes:
     - description: Fix aidmaster transform retention policy by adding a dedicated `_last_seen` timestamp field that survives the destination pipeline field strip.

packages/crowdstrike/data_stream/fdr/_dev/test/scripts/aidmaster_lookup_join.txt

@@ -0,0 +1,123 @@
+# Validate the aidmaster destination pipeline, lookup index, and LOOKUP JOIN.
+#
+# Indexes pre-processed FDR documents (as they would appear after the default
+# ingest pipeline) directly into the data stream, then pushes the
+# aidmaster doc through the destination pipeline into the lookup index,
+# and verifies the lookup is populated and queryable via ES|QL LOOKUP JOIN.
+#
+# The transform checkpoint is non-deterministic across re-runs (Fleet
+# preserves it when fleet_transform_version is unchanged), so instead of
+# waiting for the continuous transform we manually exercise the destination
+# pipeline. The transform's source query and scheduling are validated by
+# the system tests.
+
+[!external_stack] skip 'Skipping external stack test.'
+[!exec:jq] skip 'Skipping test requiring absent jq command'
+[!exec:curl] skip 'Skipping test requiring absent curl command'
+
+use_stack -profile ${CONFIG_PROFILES}/${PROFILE}
+
+# Install the package so transforms, pipelines, and index templates are created.
+add_package -profile ${CONFIG_PROFILES}/${PROFILE}
+
+# Index an aidmaster event into the FDR data stream. pipeline=_none
+# bypasses the FDR ingest pipeline (already tested by pipeline tests).
+exec curl -s -u elastic:changeme --cacert ${CONFIG_PROFILES}/${PROFILE}/certs/ca-cert.pem -X POST 'https://localhost:9200/logs-crowdstrike.fdr-default/_doc?refresh=true&pipeline=_none' -H 'Content-Type: application/json' -d @aidmaster_event.json
+stdout '"result":"created"'
+
+# Index a data event (ProcessRollup2) for the same host.
+exec curl -s -u elastic:changeme --cacert ${CONFIG_PROFILES}/${PROFILE}/certs/ca-cert.pem -X POST 'https://localhost:9200/logs-crowdstrike.fdr-default/_doc?refresh=true&pipeline=_none' -H 'Content-Type: application/json' -d @data_event.json
+stdout '"result":"created"'
+
+# Push the aidmaster doc through the destination pipeline directly
+# into the lookup index. This exercises the field renames, host.id
+# preservation, and keep logic without depending on transform checkpoints.
+exec curl -s -u elastic:changeme --cacert ${CONFIG_PROFILES}/${PROFILE}/certs/ca-cert.pem -X POST 'https://localhost:9200/crowdstrike_lookup.dest_aidmaster-1/_doc?refresh=true&pipeline='${CURRENT_VERSION}'-aidmaster_lookup_namespaced' -H 'Content-Type: application/json' -d @aidmaster_event.json
+stdout '"result":"created"'
+
+# Verify the lookup index is populated.
+exec curl -s -u elastic:changeme --cacert ${CONFIG_PROFILES}/${PROFILE}/certs/ca-cert.pem -X GET 'https://localhost:9200/crowdstrike_lookup.aidmaster/_count'
+cp stdout count.json
+exec jq -r '.count' count.json
+stdout '^[1-9]'
+
+# Verify the lookup document has the expected fields.
+exec curl -s -u elastic:changeme --cacert ${CONFIG_PROFILES}/${PROFILE}/certs/ca-cert.pem -X GET 'https://localhost:9200/crowdstrike_lookup.aidmaster/_search?size=10'
+cp stdout lookup_docs.json
+
+# Check the host.id is preserved.
+exec jq -r '.hits.hits[]._source.host.id' lookup_docs.json
+stdout '443de0bbc349316f0d394439c57beaba'
+
+# Check the hostname was renamed into the crowdstrike.info.host namespace.
+exec jq -r '.hits.hits[]._source.crowdstrike.info.host.hostname' lookup_docs.json
+stdout 'TESTHOST01'
+
+# Check the cid carried over under the crowdstrike.info.host namespace.
+exec jq -r '.hits.hits[]._source.crowdstrike.info.host.cid' lookup_docs.json
+stdout 'test-cid-000000000000000000000000'
+
+# Check _last_seen was populated (retention field).
+exec jq -r '.hits.hits[]._source.crowdstrike.info.host._last_seen' lookup_docs.json
+stdout '2025-04-01'
+
+# Run an ES|QL LOOKUP JOIN query to verify end-to-end enrichment.
+# LOOKUP JOIN requires ES 8.16+; alias resolution in LOOKUP JOIN
+# requires 8.19+. If the stack is too old the query returns an error
+# instead of a values array.
+exec curl -s -u elastic:changeme --cacert ${CONFIG_PROFILES}/${PROFILE}/certs/ca-cert.pem -X POST 'https://localhost:9200/_query' -H 'Content-Type: application/json' -d @esql_query.json
+cp stdout esql_result.json
+stdout '"values"'
+
+# The LOOKUP JOIN result should contain the enriched hostname.
+exec jq -r '[.values[][]] | map(select(. == "TESTHOST01")) | first' esql_result.json
+stdout 'TESTHOST01'
+
+-- aidmaster_event.json --
+{
+  "@timestamp": "2025-04-01T12:00:00.000Z",
+  "event": {
+    "action": "AIDMaster",
+    "category": ["host"],
+    "kind": "metric",
+    "type": ["info"],
+    "ingested": "2025-04-01T12:01:00.000Z"
+  },
+  "host": {
+    "id": "443de0bbc349316f0d394439c57beaba",
+    "hostname": "TESTHOST01",
+    "name": "TESTHOST01",
+    "domain": "TESTDOMAIN",
+    "os": { "type": "windows", "version": "10.0.19041" }
+  },
+  "observer": { "version": "7.20.16407.0" },
+  "crowdstrike": {
+    "cid": "test-cid-000000000000000000000000"
+  },
+  "log": {
+    "file": { "path": "/var/log/falcon/aidmaster" }
+  }
+}
+-- data_event.json --
+{
+  "@timestamp": "2025-04-01T12:05:00.000Z",
+  "event": {
+    "action": "ProcessRollup2",
+    "category": ["process"],
+    "kind": "event",
+    "type": ["start"],
+    "ingested": "2025-04-01T12:06:00.000Z"
+  },
+  "host": {
+    "id": "443de0bbc349316f0d394439c57beaba",
+    "os": { "type": "windows" }
+  },
+  "crowdstrike": {
+    "cid": "test-cid-000000000000000000000000",
+    "event_simpleName": "ProcessRollup2"
+  }
+}
+-- esql_query.json --
+{
+  "query": "FROM logs-crowdstrike.fdr-default | WHERE event.action == \"ProcessRollup2\" | LOOKUP JOIN crowdstrike_lookup.aidmaster ON host.id | KEEP event.action, host.id, crowdstrike.info.host.hostname | LIMIT 10"
+}

packages/crowdstrike/data_stream/fdr/_dev/test/scripts/userinfo_lookup_join.txt

@@ -32,17 +32,17 @@ stdout '"result":"created"'
 # Push the UserIdentity doc through the destination pipeline directly
 # into the lookup index. This exercises the field renames, composite key
 # construction, and keep logic without depending on transform checkpoints.
-exec curl -s -u elastic:changeme --cacert ${CONFIG_PROFILES}/${PROFILE}/certs/ca-cert.pem -X POST 'https://localhost:9200/logs-crowdstrike_lookup.dest_userinfo-1/_doc?refresh=true&pipeline='${CURRENT_VERSION}'-userinfo_lookup_namespaced' -H 'Content-Type: application/json' -d @useridentity_event.json
+exec curl -s -u elastic:changeme --cacert ${CONFIG_PROFILES}/${PROFILE}/certs/ca-cert.pem -X POST 'https://localhost:9200/crowdstrike_lookup.dest_userinfo-1/_doc?refresh=true&pipeline='${CURRENT_VERSION}'-userinfo_lookup_namespaced' -H 'Content-Type: application/json' -d @useridentity_event.json
 stdout '"result":"created"'
 
 # Verify the lookup index is populated.
-exec curl -s -u elastic:changeme --cacert ${CONFIG_PROFILES}/${PROFILE}/certs/ca-cert.pem -X GET 'https://localhost:9200/logs-crowdstrike_lookup.userinfo/_count'
+exec curl -s -u elastic:changeme --cacert ${CONFIG_PROFILES}/${PROFILE}/certs/ca-cert.pem -X GET 'https://localhost:9200/crowdstrike_lookup.userinfo/_count'
 cp stdout count.json
 exec jq -r '.count' count.json
 stdout '^[1-9]'
 
 # Verify the lookup document has the expected fields.
-exec curl -s -u elastic:changeme --cacert ${CONFIG_PROFILES}/${PROFILE}/certs/ca-cert.pem -X GET 'https://localhost:9200/logs-crowdstrike_lookup.userinfo/_search?size=10'
+exec curl -s -u elastic:changeme --cacert ${CONFIG_PROFILES}/${PROFILE}/certs/ca-cert.pem -X GET 'https://localhost:9200/crowdstrike_lookup.userinfo/_search?size=10'
 cp stdout lookup_docs.json
 
 # Check the synthetic composite key.
@@ -126,5 +126,5 @@ stdout 'jane.doe'
 }
 -- esql_query.json --
 {
-  "query": "FROM logs-crowdstrike.fdr-default | WHERE event.action == \"ProcessRollup2\" | EVAL host_user_key = CONCAT(host.id, \"::\", user.id) | LOOKUP JOIN logs-crowdstrike_lookup.userinfo ON host_user_key | KEEP event.action, user.id, crowdstrike.info.user.name | LIMIT 10"
+  "query": "FROM logs-crowdstrike.fdr-default | WHERE event.action == \"ProcessRollup2\" | EVAL host_user_key = CONCAT(host.id, \"::\", user.id) | LOOKUP JOIN crowdstrike_lookup.userinfo ON host_user_key | KEEP event.action, user.id, crowdstrike.info.user.name | LIMIT 10"
 }