Changes related to merging RFC 0007 at stage 3 #1066

webmat · 2020-10-30T20:38:24Z

Effects the changes from #1017. To be merged at the same time, or after.

Preview the new user "usage" page:

TODO

Categorization section
Notes about reuse (as a different name) of user fields within user. Namely that these don't get carried around when nesting user fields elsewhere.
Add a section about deprecating usage of host.user.*
I'm also considering adding a TOC at the top of this page, to give folks a sense of the sections contained within this humongous page.
Wait for RFC stage 3 to be merged [RFC] Multiple users in an event, stage 3 PR #1017
Identify these new field reuses as beta
- Wait for Beta label support #1051 to be merged

Questions

Towards the end of this page, I link to the RFC in the ECS repo, for folks who want to see mapping examples. I'm trying to avoid making this page 2x longer still. But will users (not ECS contributors) be lost, when they land in the RFC?
Allow custom descriptions for self-nesting reuses #1174 Out of scope for this PR, but I think that the listing of user fields in the field reuse section is a bit off. Perhaps we should mention that these are actually the user fields (right now it's designed to list other field sets nested here), or perhaps we should redesign this a bit. Note that this also affects the process.parent reuse.

webmat · 2020-10-30T20:58:58Z

This is not the only thing in this PR, but here's the preview of the new user usage page

webmat · 2020-11-02T21:23:28Z

This is ready for a review.

The "subtleties" section makes me want to cry a little 😢 😂

ebeahan · 2020-11-05T19:14:44Z

Responding to the open questions:

Towards the end of this page, I link to the RFC in the ECS repo, for folks who want to see mapping examples. I'm trying to avoid making this page 2x longer still. But will users (not ECS contributors) be lost, when they land in the RFC?

I agree it could cause some confusion, but I think the confusion would be minor. Perhaps we can adjust the wording in the link from "RFC" to something more general like "feature proposal", for users not familiar with an RFC document?

The concern over page length is understandable, but I can also see the mappings being a useful reference for users. With the sample mappings external to the docs, it might make discoverability more challenging (an extra click, not searchable from the docs site).

Out of scope for this PR, but I think that the listing of user fields in the field reuse section is a bit off. Perhaps we should mention that these are actually the user fields (right now it's designed to list other field sets nested here), or perhaps we should redesign this a bit. Note that this also affects the process.parent reuse.

I made this observation too recently. In #1051, I'm proposing introducing a beta attribute to the reusable expected object. Perhaps we can also consider adding a secondary description attribute as well?

ebeahan · 2020-11-05T19:58:17Z

The "subtleties" section makes me want to cry a little

Yes this is a challenging area to detail!

I don't think it's necessary for this PR, but in the future perhaps we can add some sort of visual as another way of explaining the concepts.

ebeahan

I added a handful of minor notes, but I think overall this is great @webmat! Having a consolidated reference is going to be a great resource!

I'll hold off on giving the final 👍 until we have the beta labeling support from #1051 finalized and the appropriate changes incorporated here.

ebeahan · 2020-11-05T22:01:57Z

docs/usage/user.asciidoc

+when there's more than one user present on the event. `user.*` fields at the root
+of the event represent the user performing the action.
+
+In many cases, events that only mention one user should populate the user fields


A high-level summary of the guidance covered could be a useful companion quick-reference to the more detailed explanations. I could see it both early in the doc as well as towards the end as a summary.

Examples of quick guidance:

Always populate user.* in addition to more purpose-specific fields when one or more users are present in an event. If more than one user, the user performing the main action populates user.*.

Any users present in the event should also be captured in related.user

etc. etc.

I feel like this is not a blocker to this PR. WDYT?

Perhaps we could note down some of the points for this quick guidance in a separate issue, and address separately?

Agreed, it's not a blocker here. We can capture and address later on 😄 .

ebeahan · 2020-11-05T22:16:37Z

docs/usage/user.asciidoc

+}
+-----------
+
+Whenever an event source populates the `client` and `server` fields in addition


I could see a second example which builds on the first, showing both client/server + source/destination fields all populated in the single event. WDYT?

Here I think I would err on the side of "terseness", if we can call such a long page terse. If the idea is to reinforce this pattern, perhaps we can instead link to the page on mapping network events?

I've added a link at the end of this paragraph

docs/usage/user.asciidoc

Co-authored-by: Eric Beahan <ebeahan@gmail.com>

This section still makes me cry a little!

webmat · 2020-12-01T15:19:16Z

@elasticmachine, run elasticsearch-ci/docs

ebeahan

The adjustments are looking great!

Towards the end of this page, I link to the RFC in the ECS repo, for folks who want to see mapping examples. I'm trying to avoid making this page 2x longer still. But will users (not ECS contributors) be lost, when they land in the RFC?

I think linking to the RFC is acceptable for curious, interested users.

I responded to the second question in my notes below.

docs/usage/user.asciidoc

docs/field-details.asciidoc

Co-authored-by: Eric Beahan <ebeahan@gmail.com>

ebeahan

Thanks for the iterations here! LGTM 👍

Conflict: deleted file rfcs/text/0007-multiple-users.md as RFCs are not backported to version branches.

* bumping version for 1.x release branch (elastic#921) * [1.x] add related.hosts (elastic#913) (elastic#924) * [1.x][DOCS] Fixes SIEM links (elastic#936) * [1.x] Consolidate field-details doc template (elastic#897) (elastic#946) * Add http.[request|response].mime_type (elastic#944) (elastic#949) * [1.x] Cut 1.6 Changelog (elastic#933) (elastic#952) (elastic#953) Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co> * [1.x] Add threat.technique.subtechnique (elastic#951) (elastic#956) Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com> * [1.x] Nest as for foreign reuse (elastic#960) (elastic#962) * [1.x] Remove `expected_event_types` from protocol (elastic#964) (elastic#965) * [1.x] Expand definitions of source and destination field sets (elastic#967) (elastic#973) * [1.x] Introduce `--strict` flag (elastic#937) (elastic#975) * [1.x] Add example value composite type checking (elastic#966) (elastic#976) * Add example value composite type checking (elastic#966) * generate csv artifact * [1.x] Add event category configuration (elastic#963) (elastic#977) * [1.x] Add normalizer multi-field capability (elastic#971) (elastic#978) Co-authored-by: Eric Beahan <ebeahan@gmail.com> Co-authored-by: Madison Caldwell <madison.rey.caldwell@gmail.com> * [1.x] Add mapping network event guidance doc (elastic#969) (elastic#983) * [1.x] Removing unneeded link under `Additional Information` (elastic#984) (elastic#985) * [1.x] Add discrete attribute to field details page headers (elastic#989) (elastic#990) * [1.x] Uniformity across domain name breakdown fields (elastic#981) (elastic#994) Co-authored-by: Mathieu Martin <webmat@gmail.com> * Add --oss flag to the ECS generator script (elastic#991) (elastic#995) * Add network directions ingress and egress (elastic#945) (elastic#997) * Mention ECS Mapper in the main documentation (elastic#987) (elastic#1000) Co-authored-by: Dan Roscigno <dan@roscigno.com> * [1.x] Introduce experimental artifacts (elastic#993) (elastic#1001) Co-authored-by: Mathieu Martin <webmat@gmail.com> * Bump version to 1.8.0-dev in branch 1.x (elastic#1011) * Cut 1.7 changelog (elastic#1010) (elastic#1012) * [1.x] Clarify that file extension should exclude the dot. (elastic#1016) (elastic#1020) * [1.x] Add usage docs section (elastic#988) (elastic#1024) Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co> * [1.x] feat: include alias path when generating template (elastic#877) (elastic#1035) Co-authored-by: Richard Gomez <32133502+rgmz@users.noreply.github.com> * [1.x] Add support for `scaling_factor` in the generator (elastic#1042) (elastic#1055) Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co> * [1.x] Add fallback for constant_keyword (elastic#1046) (elastic#1056) Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co> * [1.x] Add wildcard type support to go code generator (elastic#1050) (elastic#1057) * add wildcard type support * also add version and constant_keyword * changelog * [1.x] New default make task that generates main and experimental artifacts. (elastic#1041) (elastic#1060) Also changing the order of the 'generate' task: it now starts with the new generator, then runs the legacy scripts. * [1.x] Change the index pattern in the sample template. (elastic#1048) (elastic#1068) * [1.x] Prepare link to Logs docs changing with the 7.10 release in "getting-started" (elastic#1073) (elastic#1079) Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> * [1.x] Prepare link to Logs docs changing with the 7.10 release in "products-solutions" page (elastic#1074) (elastic#1083) Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> * [1.x] Add event.category session. (elastic#1049) (elastic#1093) Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co> * [1.x] Add event.category registry (elastic#1040) (elastic#1094) Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co> * [1.x] Add --ref support for experimental artifacts (elastic#1063) (elastic#1101) Co-authored-by: Mathieu Martin <webmat@gmail.com> * [1.x] Remove experimental event.original definition (elastic#1053) (elastic#1104) * [1.x] Add missing `process.thread.name` to experimental definitions (elastic#1103) (elastic#1106) * [1.x] Remove index parameter for wildcard fields (elastic#1115) (elastic#1119) * [1.x] Add dns.answer object into experimental schema (elastic#1118) (elastic#1121) * [1.x] Clarify x509 definition guidance for network events with only one cert (elastic#1114) (elastic#1123) * [1.x] Indicate when artifacts include experimental changes (elastic#1117) (elastic#1125) * [1.x] Add os.type field, with list of allowed values (elastic#1111) (elastic#1130) * [1.x] Add support for constant_keyword's 'value' parameter (elastic#1112) (elastic#1132) * [1.x] Beta label support (elastic#1051) (elastic#1133) Co-authored-by: Mathieu Martin <webmat@gmail.com> * [1.x] Backport elastic#1134 and elastic#1135 (elastic#1136) * Remove temporary ifeval in "getting started" page, add link to Metrics docs (elastic#1134) * Remove temporary ifeval from products page, add link to Metrics (elastic#1135) * Two small documentation backports (elastic#1149) * Remove an incorrect `event.type` from the 'converting' page (elastic#1146) * Mention Logstash support for ECS in the 'products' page (elastic#1147) * [1.x] Reinforce the exclusion of the leading dot from url.extension (elastic#1151) (elastic#1152) * [1.x] Make all fields linkable directly via an HTML ID (elastic#1148) (elastic#1154) * [1.x] Tracing fields should be at the root (elastic#1165) * Add notice to the tracing field set, about not nesting field names. (elastic#1162) * Tracing fields should be at top level in Beats artifact (elastic#1164) * [1.x] Usage of brackets for a URL containing IPv6 address (elastic#1131) (elastic#1168) * [1.x] 6.x index template data type fallback (elastic#1171) (elastic#1172) * [1.x] Apply RFC 0007 stage 3 changes - multi-user (elastic#1066) (elastic#1175) Conflict: deleted file rfcs/text/0007-multiple-users.md as RFCs are not backported to version branches. * [1.x] Handle `error.stack_trace` case for ES 6.x template (elastic#1176) (elastic#1177) * [1.x] Add composable index templates artifacts (elastic#1156) (elastic#1179) * [1.x] Move _meta section back inside mappings, in legacy templates. (elastic#1186) (elastic#1187) Backports the following commits to 1.x: * Move _meta section back inside mappings, in legacy templates. (elastic#1186) This fixes an issue introduced by elastic#1156, discovered in elastic#1180. Composable templates support `_meta` at the template's root, but legacy templates don't. So we're just putting it back inside the mappings for legacy templates. This also fixes missing updates to the component template, after the introduction of wildcard in elastic#1098. * [1.x] Apply the RFC 0005 stage 2 (host metrics) changes in the experimental artifacts (elastic#1159) (elastic#1184) Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co> * [1.x] Stage 3 changes for wildcard RFC 0001 (elastic#1098) (elastic#1183) * [1.x] Conditional handling in es_template.template_settings (elastic#1191) (elastic#1192) * [1.x] Artifacts docs page (elastic#1189) (elastic#1195) * [1.x] Remove beta warning label from categorization fields docs (elastic#1067) (elastic#1196) * [1.x] Correct wording of `event.reference` description (elastic#1181) (elastic#1197) * Bump version to 1.9.0-dev in branch 1.x (elastic#1198) * [1.x] Cut 1.8 FF changelog.next.md elastic#1199 (elastic#1201) * Merge custom and core multi_fields arrays (elastic#982) (elastic#1213) Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com> * [1.x] Stage 2 changes for RFC 0009 - data_stream fields (elastic#1215) (elastic#1222) * [1.x] add http.request.id (elastic#1208) (elastic#1223) Co-authored-by: Eric Beahan <eric.beahan@elastic.co> Co-authored-by: Gil Raphaelli <gil@elastic.co> * [1.x] add cloud.service.name (elastic#1204) (elastic#1224) * add cloud.platform * expand cloud.platform description * move to cloud.service.name Co-authored-by: Gil Raphaelli <gil@elastic.co> * [1.x] Add ssdeep hash (elastic#1169) (elastic#1227) Co-authored-by: Andrew Stucki <andrew.stucki@elastic.co> * [CI] Switch to GitHub actions (elastic#1236) (elastic#1245) Co-authored-by: Eric Beahan <ebeahan@gmail.com> Co-authored-by: Andrew Stucki <andrew.stucki@elastic.co> * Revert wildcard adoption back to experimental stage (elastic#1235) (elastic#1243) * Add scaled_float type to go generator (elastic#1250) (elastic#1251) * add scaled_float * changelog * Add categorization fields usage docs (elastic#1242) (elastic#1257) * add time_zone, postal_code, and continent_code (elastic#1229) (elastic#1258) * Specify MAC address format (elastic#456) (elastic#1260) Co-authored-by: Robin Schneider <36660054+ypid-geberit@users.noreply.github.com> * finalize 1.8.0 changelog (elastic#1262) (elastic#1265) * Add additional host fields (elastic#1248) (elastic#1267) Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Stage 1 changes for RFC 0014 - extend pe fields (elastic#1256) (elastic#1270) * Add 2 fields to code_signature (elastic#1269) (elastic#1272) Co-authored-by: Yamin Tian <56367679+Trinity2019@users.noreply.github.com> * Stage 3 changes for RFC 0007 - remove beta attribute (elastic#1271) (elastic#1273) * Stage 1 experimental changes for RFC 0008 - threat.indicator fields (elastic#1268) (elastic#1274) * Stage 1 changes for RFC 0015 - add elf fieldset (elastic#1261) (elastic#1275) * Cut 1.9 FF CHANGELOG.next.md (elastic#1277) * lock go version in actions (elastic#1283) (elastic#1290) * Bump jinja2 from 2.11.2 to 2.11.3 in /scripts (elastic#1310) (elastic#1320) * Bump jinja2 from 2.11.2 to 2.11.3 in /scripts * Bump pyyaml from 5.3b1 to 5.4 in /scripts (elastic#1318) (elastic#1325) Co-authored-by: Eric Beahan <eric.beahan@elastic.co> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Adjust terminology - change whitelist to allowlist (elastic#1315) (elastic#1331) Co-authored-by: Dominic Page <11043991+djptek@users.noreply.github.com> * Remove -dev label from 1.9 version (elastic#1329) * remove -dev label from 1.9 version * generate artifacts * removing rules artifacts * Cut 1.9 changelog (elastic#1328) * move 1.9 changes to changelog * add 1.9 release changes

Mathieu Martin added 2 commits October 30, 2020 11:10

Apply new user field reuse in main ECS field definitions.

746c156

WIP transferring a lot of the RFC explanations to a new user usage page

c124d82

webmat self-assigned this Oct 30, 2020

webmat added 1.x needs_backport labels Oct 30, 2020

Changelog

613a525

webmat mentioned this pull request Oct 30, 2020

[RFC] Multiple users in an event, stage 3 PR #1017

Merged

6 tasks

webmat mentioned this pull request Nov 2, 2020

Lack of clarity on how the trace fields are supposed to be used #998

Open

Mathieu Martin added 5 commits November 2, 2020 14:42

Section on categorizing IAM activity

6906487

Add a TOC to the page

6d805b6

Subtleties around field reuse

4efecc9

Deprecation of host.user.*

0f247f2

Additional callouts for one of the examples

7002fe4

webmat marked this pull request as ready for review November 2, 2020 20:56

webmat requested a review from ebeahan November 2, 2020 21:02

Fix list nesting, improve wording in a few places

daafe57

ebeahan reviewed Nov 5, 2020

View reviewed changes

webmat and others added 4 commits November 6, 2020 08:28

escalate "to"

63b203f

Co-authored-by: Eric Beahan <ebeahan@gmail.com>

Root uid is zeero

65e534b

Clarify instead of what

f82ba40

Co-authored-by: Eric Beahan <ebeahan@gmail.com>

Add a short section on user identifier gotchas.

459976e

Another attempt at explaining the self-reuse subtleties.

7c3b299

This section still makes me cry a little!

ebeahan added the ready Issues we'd like to address in the future. label Nov 17, 2020

webmat added the 1.8.0 label Nov 17, 2020

Mathieu Martin added 2 commits November 23, 2020 16:43

Merge branch 'master' into rfc/0007/apply-stage-3

f1db3b0

Introduce the 'beta' markers for the new 'user' field reuses

dbef721

ebeahan reviewed Dec 4, 2020

View reviewed changes

docs/usage/user.asciidoc Show resolved Hide resolved

docs/usage/user.asciidoc Outdated Show resolved Hide resolved

docs/field-details.asciidoc Show resolved Hide resolved

webmat and others added 2 commits December 4, 2020 14:49

Break a paragraph in two, to make sure that second part stands out

e573f8e

Co-authored-by: Eric Beahan <ebeahan@gmail.com>

Link to 'Mapping Network Events' page, as suggested in review

94cac00

ebeahan approved these changes Dec 7, 2020

View reviewed changes

webmat merged commit 5f5c4ec into elastic:master Dec 7, 2020

webmat pushed a commit to webmat/ecs that referenced this pull request Dec 7, 2020

Apply RFC 0007 stage 3 changes - multi-user (elastic#1066)

f60a306

Conflict: deleted file rfcs/text/0007-multiple-users.md as RFCs are not backported to version branches.

webmat mentioned this pull request Dec 7, 2020

[1.x] Apply RFC 0007 stage 3 changes - multi-user (#1066) #1175

Merged

webmat pushed a commit that referenced this pull request Dec 7, 2020

[1.x] Apply RFC 0007 stage 3 changes - multi-user (#1066) (#1175)

48e1ddc

Conflict: deleted file rfcs/text/0007-multiple-users.md as RFCs are not backported to version branches.

webmat removed the needs_backport label Dec 7, 2020

This was referenced May 5, 2021

Upstream ECS merge of v1.9.0 dseeley/ecs#1

Closed

Upstream ECS merge of v1.9.0 dseeley/ecs#2

Closed

Upstream ECS merge of v1.9.0 dseeley/ecs#3

Closed

Upstream ECS merge of v1.9.0 dseeley/ecs#4

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes related to merging RFC 0007 at stage 3 #1066

Changes related to merging RFC 0007 at stage 3 #1066

webmat commented Oct 30, 2020 •

edited

Loading

webmat commented Oct 30, 2020

webmat commented Nov 2, 2020

ebeahan commented Nov 5, 2020

ebeahan commented Nov 5, 2020

ebeahan left a comment

ebeahan Nov 5, 2020

webmat Nov 6, 2020

webmat Dec 4, 2020

ebeahan Dec 4, 2020

ebeahan Nov 5, 2020

webmat Nov 6, 2020

webmat Dec 4, 2020

webmat commented Dec 1, 2020

ebeahan left a comment

ebeahan left a comment

Changes related to merging RFC 0007 at stage 3 #1066

Changes related to merging RFC 0007 at stage 3 #1066

Conversation

webmat commented Oct 30, 2020 • edited Loading

webmat commented Oct 30, 2020

webmat commented Nov 2, 2020

ebeahan commented Nov 5, 2020

ebeahan commented Nov 5, 2020

ebeahan left a comment

Choose a reason for hiding this comment

ebeahan Nov 5, 2020

Choose a reason for hiding this comment

webmat Nov 6, 2020

Choose a reason for hiding this comment

webmat Dec 4, 2020

Choose a reason for hiding this comment

ebeahan Dec 4, 2020

Choose a reason for hiding this comment

ebeahan Nov 5, 2020

Choose a reason for hiding this comment

webmat Nov 6, 2020

Choose a reason for hiding this comment

webmat Dec 4, 2020

Choose a reason for hiding this comment

webmat commented Dec 1, 2020

ebeahan left a comment

Choose a reason for hiding this comment

ebeahan left a comment

Choose a reason for hiding this comment

webmat commented Oct 30, 2020 •

edited

Loading