Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

o365: fix mappings for dynamically mapped fields #7988

Merged
merged 1 commit into from Sep 28, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/o365/changelog.yml
@@ -1,4 +1,9 @@
# newer versions go on top
- version: 1.24.1
changes:
- description: Fix mappings for dynamically mapped fields.
type: enhancement
link: https://github.com/elastic/integrations/pull/1
- version: 1.24.0
changes:
- description: ECS version updated to 8.10.0.
Expand Down
22 changes: 20 additions & 2 deletions packages/o365/data_stream/audit/fields/fields.yml
Expand Up @@ -34,6 +34,9 @@
type: keyword
- name: ExchangeMetaData.*
type: object
# This object can also contain date fields, but we cannot express multiple dynamic mapping types here.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This object can also contain date fields

For these cases I assume the data has always been mapped as keyword because Fleet using "date_detection": false in templates. So this isn't a regression.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, this is just a note for why it is.

object_type: long
object_type_mapping_type: long
- name: Category
type: keyword
- name: ClientAppId
Expand Down Expand Up @@ -68,8 +71,14 @@
type: keyword
- name: ExceptionInfo.*
type: object
# This should be boolean→boolean falling back to *→keyword, but this is
# not expressible here; object_type_mapping_type cannot be 'boolean'.
object_type: keyword
object_type_mapping_type: '*'
- name: ExtendedProperties.*
type: object
object_type: keyword
object_type_mapping_type: '*'
- name: ExternalAccess
type: boolean
- name: FileSizeBytes
Expand All @@ -90,8 +99,12 @@
type: keyword
- name: Item.*
type: object
object_type: keyword
object_type_mapping_type: '*'
Comment on lines 100 to +103
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the dynamic templates, it looks like this is not necessary.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why didn't it show up in the dynamic_templates?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's why I'm confused.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe in a bug in package-spec?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suspect Fleet. Like perhaps the dedupFields function (https://github.com/elastic/kibana/blob/ed8225f7bcb9f8d16b2241c8de26cd8103fcb942/x-pack/plugins/fleet/server/services/epm/fields/field.ts#L124). Given that this would appear to be a parent of the Item.*.* I wonder if it is being ignored.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logged an issue at elastic/kibana#167553.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can move forward without this being in the template because it's only purpose was to satisfy the field validation (all fields found in test cases must be defined) within elastic-package and it still does. The default dynamic mappings will work fine.

- name: Item.*.*
type: object
object_type: keyword
object_type_mapping_type: '*'
- name: ItemName
type: keyword
- name: ItemType
Expand All @@ -118,10 +131,10 @@
type: keyword
- name: Members
type: flattened
- name: Members.*
type: object
- name: ModifiedProperties.*.*
type: object
object_type: keyword
object_type_mapping_type: '*'
- name: Name
type: keyword
- name: NewValue
Expand All @@ -138,6 +151,8 @@
type: keyword
- name: Parameters.*
type: object
object_type: keyword
object_type_mapping_type: '*'
- name: PolicyDetails
type: flattened
- name: PolicyId
Expand All @@ -150,6 +165,9 @@
type: boolean
- name: SharePointMetaData.*
type: object
# This object may contain date formatted fields, but we do not ensure validity, so leave as keyword.
object_type: keyword
object_type_mapping_type: '*'
- name: SessionId
type: keyword
- name: Severity
Expand Down
1 change: 0 additions & 1 deletion packages/o365/docs/README.md
Expand Up @@ -279,7 +279,6 @@ An example event for `audit` looks as following:
| o365.audit.MailboxOwnerSid | | keyword |
| o365.audit.MailboxOwnerUPN | | keyword |
| o365.audit.Members | | flattened |
| o365.audit.Members.\* | | object |
| o365.audit.ModifiedProperties.\*.\* | | object |
| o365.audit.Name | | keyword |
| o365.audit.NewValue | | keyword |
Expand Down
2 changes: 1 addition & 1 deletion packages/o365/manifest.yml
@@ -1,6 +1,6 @@
name: o365
title: Microsoft 365
version: "1.24.0"
version: "1.24.1"
description: Collect logs from Microsoft 365 with Elastic Agent.
type: integration
format_version: "3.0.0"
Expand Down