okta: allow user configuration of debug_data flattened use #9868

efd6 · 2024-05-15T03:56:37Z

Status: Blocked by elastic/kibana#183496

Proposed commit message

See title.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [okta.system] Utilize 'subobjects: false' for debugContext.debugData #9863

Screenshots

efd6 · 2024-05-15T04:00:37Z

This change does not currently pass.

--- Test results for package: okta - START ---
FAILURE DETAILS:
okta/system test-okta-system-no-flattened-events.log:
[0] field "okta.debug_context.debug_data.behaviors.new_country" is undefined
[1] field "okta.debug_context.debug_data.behaviors.new_device" is undefined
[2] field "okta.debug_context.debug_data.behaviors.new_geo_location" is undefined
[3] field "okta.debug_context.debug_data.behaviors.new_ip" is undefined
[4] field "okta.debug_context.debug_data.behaviors.new_state" is undefined
[5] field "okta.debug_context.debug_data.behaviors.velocity" is undefined
[6] field "okta.debug_context.debug_data.behaviors.velocity_behavior" is undefined
[7] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_city" is undefined
[8] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_country" is undefined
[9] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_device" is undefined
[10] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_geo_location" is undefined
[11] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_ip" is undefined
[12] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_state" is undefined
[13] field "okta.debug_context.debug_data.log_only_security_data.behaviors.velocity" is undefined
[14] field "okta.debug_context.debug_data.log_only_security_data.risk.level" is undefined
[15] field "okta.debug_context.debug_data.log_only_security_data.risk.reasons" is undefined
[16] field "okta.debug_context.debug_data.original_principal.alternate_id" is undefined
[17] field "okta.debug_context.debug_data.original_principal.display_name" is undefined
[18] field "okta.debug_context.debug_data.original_principal.id" is undefined
[19] field "okta.debug_context.debug_data.original_principal.type" is undefined
[20] field "okta.debug_context.debug_data.prompting_policy_types" is undefined
[21] field "okta.debug_context.debug_data.risk.level" is undefined
[22] field "okta.debug_context.debug_data.risk.reasons" is undefined
[23] field "okta.debug_context.debug_data.risk_object" is undefined



╭─────────┬─────────────┬───────────┬───────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                 │ RESULT                                                                      │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────┼──────────────┤
│ okta    │ system      │ pipeline  │ test-okta-system-events.log               │ PASS                                                                        │  17.328337ms │
│ okta    │ system      │ pipeline  │ test-okta-system-no-flattened-events.log  │ FAIL: test case failed: one or more problems with fields found in documents │  34.252526ms │
│ okta    │ system      │ pipeline  │ test-okta-system-yes-flattened-events.log │ PASS                                                                        │  28.417599ms │
│ okta    │ system      │ pipeline  │ (ingest pipeline warnings)                │ PASS                                                                        │ 395.535137ms │
╰─────────┴─────────────┴───────────┴───────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────┴──────────────╯

So add one of these to fix this…

diff --git a/packages/okta/data_stream/system/fields/fields.yml b/packages/okta/data_stream/system/fields/fields.yml
index b0f51b1b0..c314f671c 100644
--- a/packages/okta/data_stream/system/fields/fields.yml
+++ b/packages/okta/data_stream/system/fields/fields.yml
@@ -273,3 +273,5 @@
   fields:
     - name: ip_chain
       type: flattened
+- name: okta.debug_context.debug_data.behaviors.new_city
+  type: keyword
\ No newline at end of file
diff --git a/packages/okta/docs/README.md b/packages/okta/docs/README.md
index 00f10e7ea..9762a1b64 100644
--- a/packages/okta/docs/README.md
+++ b/packages/okta/docs/README.md
@@ -304,6 +304,7 @@ An example event for `system` looks as following:
 | okta.client.zone | The zone information of the client. | keyword |
 | okta.debug_context.debug_data |  | object |
 | okta.debug_context.debug_data.authn_request_id | The authorization request ID. | keyword |
+| okta.debug_context.debug_data.behaviors.new_city |  | keyword |
 | okta.debug_context.debug_data.device_fingerprint | The fingerprint of the device. | keyword |
 | okta.debug_context.debug_data.dt_hash | The device token hash | keyword |
 | okta.debug_context.debug_data.factor | The factor used for authentication. | keyword |

This however causes a mapping failure:

Error: error running package system tests: could not complete test run: can't install the package: there was an apply error: installation failed: can't install the package: could not zip-install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"mapper_parsing_exception\n\tCaused by:\n\t\tmapper_parsing_exception: Tried to add subobject [behaviors] to object [debug_data] which does not support subobjects\n\tRoot causes:\n\t\tmapper_parsing_exception: Tried to add subobject [behaviors] to object [debug_data] which does not support subobjects"}

mapper_parsing_exception
	Caused by:
		mapper_parsing_exception: Tried to add subobject [behaviors] to object [debug_data] which does not support subobjects
	Root causes:
		mapper_parsing_exception: Tried to add subobject [behaviors] to object [debug_data] which does not support subobjects

A minimal version of this is here:

- name: grandparent
  type: object
  object_type: keyword
  object_type_mapping_type: "*"
  subobjects: false
- name: grandparent.parent.child
  type: keyword

elasticmachine · 2024-05-15T04:51:47Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2024-05-15T04:59:16Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

packages/okta/data_stream/system/fields/fields.yml

packages/okta/data_stream/system/elasticsearch/ingest_pipeline/no_use_flattened_debug.yml

packages/okta/data_stream/system/manifest.yml

andrewkroh · 2024-05-21T00:22:07Z

I was testing the upgrade process going from 2.9.0 to 2.10.0 (751ce10) where I already had some data indexed, and the upgrade (using 8.13.4) fails with

mapper_exception: the [subobjects] parameter can't be updated for the object mapping [okta.debug_context.debug_data]

@zmoog @ruflin Is this the expected behavior? Can we adopt subobjects: false for an existing field?

The subobjects documentation says

The subobjects setting for existing fields and the top-level mapping definition cannot be updated.

but shouldn't Fleet be doing a rollover?

andrewkroh · 2024-05-21T04:53:21Z

packages/okta/manifest.yml

@@ -138,6 +138,13 @@ policy_templates:
            multi: false
            required: false
            show_user: true
+          - name: remove_flattened_debug


remove_flattened_debug is declared in two places. I think one of them needs to be removed.

Relates: elastic/package-spec#421

elastic-sonarqube · 2024-05-21T06:17:42Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-05-21T06:18:06Z

💚 Build Succeeded

Buildkite Build
Commit: 4604582

History

💚 Build #11621 succeeded 751ce10
💔 Build #11620 failed 720cfe8
💔 Build #11613 failed 7c922b5
💚 Build #11454 succeeded 283d7b3
💔 Build #11453 failed ee6e3f0

cc @efd6

zmoog · 2024-05-21T06:25:56Z

Is this the expected behavior? Can we adopt subobjects: false for an existing field?

The subobjects documentation says

The subobjects setting for existing fields and the top-level mapping definition cannot be updated.

but shouldn't Fleet be doing a rollover?

Yes, Fleet should do a rollover.

@flash1293 mentioned this issue during our sync, a couple of days ago. We are looking into this.

flash1293 · 2024-05-21T06:56:24Z

Sorry for this hurdle, the fleet team is already working on it here: elastic/kibana#183496

efd6 added enhancement New feature or request Integration:Okta Team:Security-Service Integrations Security Service Integrations Team labels May 15, 2024

efd6 self-assigned this May 15, 2024

efd6 added 2 commits May 15, 2024 13:27

okta: allow user configuration of debug_data flattened use

8163176

okta: clean up field definitions

ee6e3f0

efd6 force-pushed the 9863-okta branch from 88f8137 to ee6e3f0 Compare May 15, 2024 03:57

okta: work around mapping

283d7b3

efd6 marked this pull request as ready for review May 15, 2024 04:59

efd6 requested a review from a team as a code owner May 15, 2024 04:59

andrewkroh reviewed May 17, 2024

View reviewed changes

packages/okta/data_stream/system/manifest.yml Outdated Show resolved Hide resolved

packages/okta/data_stream/system/manifest.yml Outdated Show resolved Hide resolved

address pr comments

720cfe8

efd6 force-pushed the 9863-okta branch from 7c922b5 to 720cfe8 Compare May 17, 2024 04:54

fix space-containing keys

751ce10

andrewkroh reviewed May 21, 2024

View reviewed changes

address pr comment

4604582

efd6 requested a review from andrewkroh May 21, 2024 05:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

okta: allow user configuration of debug_data flattened use #9868

okta: allow user configuration of debug_data flattened use #9868

efd6 commented May 15, 2024 •

edited by andrewkroh

efd6 commented May 15, 2024 •

edited

elasticmachine commented May 15, 2024

elasticmachine commented May 15, 2024

andrewkroh commented May 21, 2024 •

edited

andrewkroh May 21, 2024

elastic-sonarqube bot commented May 21, 2024

elasticmachine commented May 21, 2024

zmoog commented May 21, 2024

flash1293 commented May 21, 2024

okta: allow user configuration of debug_data flattened use #9868

Are you sure you want to change the base?

okta: allow user configuration of debug_data flattened use #9868

Conversation

efd6 commented May 15, 2024 • edited by andrewkroh

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

efd6 commented May 15, 2024 • edited

elasticmachine commented May 15, 2024

🚀 Benchmarks report

elasticmachine commented May 15, 2024

andrewkroh commented May 21, 2024 • edited

andrewkroh May 21, 2024

Choose a reason for hiding this comment

elastic-sonarqube bot commented May 21, 2024

Quality Gate passed

elasticmachine commented May 21, 2024

💚 Build Succeeded

History

zmoog commented May 21, 2024

flash1293 commented May 21, 2024

efd6 commented May 15, 2024 •

edited by andrewkroh

efd6 commented May 15, 2024 •

edited

andrewkroh commented May 21, 2024 •

edited