Feat/olm endpoint action responses update schemas epic 1279 #198

ashokaditya · 2021-09-22T17:27:11Z

Change Summary

Sample values

.logs-endpoint.actions data stream

expand mapping

{
  "mappings": {
    "dynamic": "false",
    "_meta": {
      "package": {
        "name": "endpoint"
      },
      "managed_by": "ingest-manager",
      "managed": true
    },
    "_data_stream_timestamp": {
      "enabled": true
    },
    "dynamic_templates": [
      {
        "strings_as_keyword": {
          "match_mapping_type": "string",
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      }
    ],
    "date_detection": false,
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "EndpointActions": {
        "properties": {
          "action_id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "data": {
            "properties": {
              "command": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "comment": {
                "type": "text"
              }
            }
          },
          "expiration": {
            "type": "date"
          },
          "input_type": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "action_id": {
        "type": "alias",
        "path": "EndpointActions.action_id"
      },
      "agent": {
        "properties": {
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "agents": {
        "type": "alias",
        "path": "agent.id"
      },
      "data": {
        "properties": {
          "command": {
            "type": "alias",
            "path": "EndpointActions.data.command"
          },
          "comment": {
            "type": "alias",
            "path": "EndpointActions.data.comment"
          }
        }
      },
      "data_stream": {
        "properties": {
          "dataset": {
            "type": "constant_keyword"
          },
          "namespace": {
            "type": "constant_keyword"
          },
          "type": {
            "type": "constant_keyword"
          }
        }
      },
      "ecs": {
        "properties": {
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "error": {
        "properties": {
          "code": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "message": {
            "type": "text"
          },
          "stack_trace": {
            "type": "keyword",
            "index": false,
            "ignore_above": 1024,
            "fields": {
              "text": {
                "type": "text"
              }
            }
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "event": {
        "properties": {
          "action": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "agent_id_status": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "category": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "created": {
            "type": "date"
          },
          "end": {
            "type": "date"
          },
          "hash": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "ingested": {
            "type": "date"
          },
          "outcome": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "start": {
            "type": "date"
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "expiration": {
        "type": "alias",
        "path": "EndpointActions.expiration"
      },
      "input_type": {
        "type": "alias",
        "path": "EndpointActions.input_type"
      },
      "type": {
        "type": "alias",
        "path": "EndpointActions.type"
      },
      "user": {
        "properties": {
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "user_id": {
        "type": "alias",
        "path": "user.id"
      }
    }
  }
}

Sample document for .logs-endpoint.action data stream:

{
  "@timestamp":"2021-09-15T08:23:33.481Z",
  "EndpointActions": {
        "action_id":"f9467233-8533-42bd-9322-9e5bd33f13ce",
        "expiration":"2021-09-24T08:23:33.481Z",
        "type":"INPUT_ACTION",
        "input_type":"endpoint",
         "data": {
          "command":"unisolate",
          "comment":"action in new data stream"
        }
  },
  "agent": {
      "id":["a32fd454-764a-424f-bf0a-d93bd93407a2","fd041791-f467-474d-bb70-5dab47004483", "db19b5b5-59b0-4a22-aebb-e96895d8fd0c"]
  },
  "user": {
    "id": "elastic"
  }
}

Sample document with an error field:

{
  "@timestamp":"2021-09-15T08:23:33.481Z",
  "EndpointActions": {
        "action_id":"f9467233-8533-42bd-9322-9e5bd33f13ce",
        "expiration":"2021-09-24T08:23:33.481Z",
        "type":"INPUT_ACTION",
        "input_type":"endpoint",
         "data": {
          "command":"unisolate",
          "comment":"action in new data stream"
        }
  },
  "agent": {
      "id":["a32fd454-764a-424f-bf0a-d93bd93407a2","fd041791-f467-474d-bb70-5dab47004483", "db19b5b5-59b0-4a22-aebb-e96895d8fd0c"]
  },
 "error":{
    "message": "request error"
  },
  "user": {
    "id": "elastic"
  }
}

Using API to add a single document to .logs-endpoint.actions data stream:

POST /.logs-endpoint.actions-default/_doc
{
  "@timestamp":"2021-09-15T08:23:33.481Z",
  "EndpointActions": {
        "action_id":"f9467233-8533-42bd-9322-9e5bd33f13ce",
        "expiration":"2021-09-24T08:23:33.481Z",
        "type":"INPUT_ACTION",
        "input_type":"endpoint",
         "data": {
          "command":"unisolate",
          "comment":"action in new data stream"
        }
  },
  "agent": {
      "id":["a32fd454-764a-424f-bf0a-d93bd93407a2","fd041791-f467-474d-bb70-5dab47004483", "db19b5b5-59b0-4a22-aebb-e96895d8fd0c"]
  },
  "user": {
    "id": "elastic"
  }
}

.logs-endpoint.action.responses data stream

expand mapping

{
  "mappings": {
    "dynamic": "false",
    "_meta": {
      "package": {
        "name": "endpoint"
      },
      "managed_by": "ingest-manager",
      "managed": true
    },
    "_data_stream_timestamp": {
      "enabled": true
    },
    "dynamic_templates": [
      {
        "strings_as_keyword": {
          "match_mapping_type": "string",
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      }
    ],
    "date_detection": false,
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "EndpointActions": {
        "properties": {
          "action_id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "completed_at": {
            "type": "date"
          },
          "data": {
            "properties": {
              "command": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "comment": {
                "type": "text"
              }
            }
          },
          "started_at": {
            "type": "date"
          },
          "status": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "data": {
        "properties": {
          "command": {
            "type": "alias",
            "path": "EndpointActions.data.command"
          },
          "comment": {
            "type": "alias",
            "path": "EndpointActions.data.comment"
          }
        }
      },
      "action_id": {
        "type": "alias",
        "path": "EndpointActions.action_id"
      },
      "agent": {
        "properties": {
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "agent_id": {
        "type": "alias",
        "path": "agent.id"
      },
      "completed_at": {
        "type": "alias",
        "path": "EndpointActions.completed_at"
      },
      "data_stream": {
        "properties": {
          "dataset": {
            "type": "constant_keyword"
          },
          "namespace": {
            "type": "constant_keyword"
          },
          "type": {
            "type": "constant_keyword"
          }
        }
      },
      "ecs": {
        "properties": {
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "error": {
        "properties": {
          "code": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "message": {
            "type": "text"
          },
          "stack_trace": {
            "type": "keyword",
            "index": false,
            "ignore_above": 1024,
            "fields": {
              "text": {
                "type": "text"
              }
            }
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "event": {
        "properties": {
          "action": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "agent_id_status": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "category": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "created": {
            "type": "date"
          },
          "end": {
            "type": "date"
          },
          "hash": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "ingested": {
            "type": "date"
          },
          "outcome": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "start": {
            "type": "date"
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "started_at": {
        "type": "alias",
        "path": "EndpointActions.started_at"
      },
      "status": {
        "type": "alias",
        "path": "EndpointActions.status"
      }
    }
  }
}

Sample document for .logs-endpoint.action.responses data stream (that the endpoint writes to):

{
  "@timestamp":"2021-09-20T10:00:00.000Z", 
  "EndpointActions": {
      "action_id":"67038ba4-563d-4790-b41b-36878f6332b2",
      "completed_at":"2021-09-20T11:54:03.153Z", 
      "started_at": "2021-09-20T11:53:33.3481Z", 
      "input_type":"endpoint",
      "data":{
        "command":"unisolate",
        "comment":"unisolate host"
      }
  },
  "agent":{
    "id":"a32fd454-764a-424f-bf0a-d93bd93407a2"
  } 
}

Sample document with an error field

{
  "@timestamp":"2021-09-20T10:00:00.000Z", 
  "EndpointActions": {
      "action_id":"67038ba4-563d-4790-b41b-36878f6332b2",
      "completed_at":"2021-09-20T11:54:03.153Z", 
      "started_at": "2021-09-20T11:53:33.3481Z", 
      "input_type":"endpoint",
      "data":{
        "command":"unisolate",
        "comment":"unisolate host"
      }
  },
 "error":{
    "message": "some error happened"
 },
  "agent":{
    "id":"a32fd454-764a-424f-bf0a-d93bd93407a2"
  } 
}

Using API to add a single document to .logs-endpoint.action.responses data stream:

POST .logs-endpoint.action.responses-default/_doc
{
  "@timestamp":"2021-09-20T10:00:00.000Z", 
  "EndpointActions": {
      "action_id":"67038ba4-563d-4790-b41b-36878f6332b2",
      "completed_at":"2021-09-20T11:54:03.153Z", 
      "started_at": "2021-09-20T11:53:33.3481Z", 
      "input_type":"endpoint",
      "data":{
        "command":"unisolate",
        "comment":"unisolate host"
      }
  },
  "agent":{
    "id":"a32fd454-764a-424f-bf0a-d93bd93407a2"
  } 
}

Release Target

v7.16

Q/A

For mapping changes:

I ran make after making the schema changes, and committed any generated files (in schema/, generated/)

in order to follow ECS convention

We are going to keep this until 8.0.0. And then remove them from base fields.

custom_subsets/elastic_endpoint/action_responses/action_responses.yaml

custom_subsets/elastic_endpoint/actions/actions.yaml

custom_schemas/custom_base.yml

custom_subsets/elastic_endpoint/actions/actions.yaml

jonathan-buttner · 2021-09-22T19:48:13Z

Thanks for the ping @ashokaditya . Since ECS leverages lowercase names for the definitions of the top level fields we chose to capitalize our fields to avoid potential conflicts. An example of a conflict would be if the ECS core team released a top level field called response. That would conflict with the response field that we're creating in this PR. Could we change the top level fields in this PR to be Request and Response? I'm not up to speed on the feature request but are these fields being used to describe the type of actions and responses that a Kibana user can send to an Endpoint? If that's the case maybe we should create a single top level field called Actions or EndpointActions or something else and nest request and response under that.

Seems like most fields in this PR don't really exist in ECS (except maybe user_id) but just an FYI, the more we can use the fields that ECS already provides the better. So when we're designing messages/events in the future just keep that in the back of your mind.

Another thing we can do is if the fields we're adding make sense within a top level field that ECS already provides (but the exact field doesn't exist yet) we can extend the ECS top level field and place the new field under Ext. We've done that in a number of places in our custom schema already.

ashokaditya · 2021-09-23T07:40:46Z

Thanks for the ping @ashokaditya . Since ECS leverages lowercase names for the definitions of the top level fields we chose to capitalize our fields to avoid potential conflicts. An example of a conflict would be if the ECS core team released a top level field called response. That would conflict with the response field that we're creating in this PR. Could we change the top level fields in this PR to be Request and Response? I'm not up to speed on the feature request but are these fields being used to describe the type of actions and responses that a Kibana user can send to an Endpoint? If that's the case maybe we should create a single top level field called Actions or EndpointActions or something else and nest request and response under that.

During the course of doing this mapping update, I did think about nesting "requests" and "responses" under a top-level field, so I'm going to go ahead and give that a try. 👍 Also, this way the mapping makes also more contextual sense.

EndpointActions:
  fields:
    request:
       fields: 
         ...
         ...
    response:
      fields:
         ...
         ...

Seems like most fields in this PR don't really exist in ECS (except maybe user_id) but just an FYI, the more we can use the fields that ECS already provides the better. So when we're designing messages/events in the future just keep that in the back of your mind.

Indeed! I'll keep that in mind.

Another thing we can do is if the fields we're adding make sense within a top level field that ECS already provides (but the exact field doesn't exist yet) we can extend the ECS top level field and place the new field under Ext. We've done that in a number of places in our custom schema already.

This is also another good idea, but I think I'll rather wait and discuss this with @pzl before trying this out.

review suggestion

custom_schemas/custom_base.yml

…schemas-epic-1279

jonathan-buttner

Looks good 👍 thanks for making the changes.

custom_schemas/custom_base.yml

custom_schemas/custom_endpoint_actions.yml

custom_subsets/elastic_endpoint/action_responses/action_responses.yaml

pzl

🐑 🚀 ?

ashokaditya added 3 commits September 22, 2021 18:58

add custom schemas for action and action responses

891bade

in order to follow ECS convention

make base level custom action and response fields as alias

3c3a6bd

We are going to keep this until 8.0.0. And then remove them from base fields.

update fields and schemas

a0e7027

ashokaditya self-assigned this Sep 22, 2021

ashokaditya requested review from jonathan-buttner, pzl and ferullo September 22, 2021 17:40

ashokaditya added the Team:Onboarding and Lifecycle Mgt label Sep 22, 2021

ashokaditya commented Sep 22, 2021

View reviewed changes

custom_subsets/elastic_endpoint/action_responses/action_responses.yaml Outdated Show resolved Hide resolved

ashokaditya commented Sep 22, 2021

View reviewed changes

custom_subsets/elastic_endpoint/actions/actions.yaml Outdated Show resolved Hide resolved

ashokaditya commented Sep 22, 2021

View reviewed changes

custom_schemas/custom_base.yml Outdated Show resolved Hide resolved

ashokaditya commented Sep 22, 2021

View reviewed changes

custom_subsets/elastic_endpoint/actions/actions.yaml Show resolved Hide resolved

ashokaditya requested a review from joeypoon September 23, 2021 07:07

update mapping to avoid field collision

6cb5324

review suggestion

ashokaditya marked this pull request as ready for review September 23, 2021 14:58

jonathan-buttner reviewed Sep 23, 2021

View reviewed changes

custom_schemas/custom_base.yml Outdated Show resolved Hide resolved

ashokaditya added 2 commits September 23, 2021 21:13

no need for "

9334585

Merge branch 'master' into feat/olm-endpoint_action_responses_update_…

c3d52da

…schemas-epic-1279

jonathan-buttner approved these changes Sep 27, 2021

View reviewed changes

add error fields to response DS

91784e5

ashokaditya commented Sep 27, 2021

View reviewed changes

custom_schemas/custom_base.yml Outdated Show resolved Hide resolved

ashokaditya added 2 commits September 27, 2021 17:33

no need for the top level alias field for error.message

0dbaa0e

unify fields so we don't define redundant mappings

cc3d234

pzl reviewed Sep 27, 2021

View reviewed changes

custom_schemas/custom_endpoint_actions.yml Outdated Show resolved Hide resolved

custom_schemas/custom_endpoint_actions.yml Outdated Show resolved Hide resolved

custom_subsets/elastic_endpoint/action_responses/action_responses.yaml Outdated Show resolved Hide resolved

ashokaditya added 3 commits September 27, 2021 20:55

add error fields to action DS

66cd5d1

cleanup

7f80424

rename alias field

b187153

ashokaditya requested a review from pzl September 27, 2021 19:14

pzl approved these changes Sep 27, 2021

View reviewed changes

ashokaditya merged commit d06e6ab into elastic:master Sep 27, 2021

ashokaditya deleted the feat/olm-endpoint_action_responses_update_schemas-epic-1279 branch September 27, 2021 19:19

ferullo mentioned this pull request Oct 4, 2021

Additional field in .fleet-actions-results document elastic/beats#28237

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/olm endpoint action responses update schemas epic 1279 #198

Feat/olm endpoint action responses update schemas epic 1279 #198

ashokaditya commented Sep 22, 2021 •

edited

Loading

jonathan-buttner commented Sep 22, 2021 •

edited

Loading

ashokaditya commented Sep 23, 2021

jonathan-buttner left a comment

pzl left a comment

Feat/olm endpoint action responses update schemas epic 1279 #198

Feat/olm endpoint action responses update schemas epic 1279 #198

Conversation

ashokaditya commented Sep 22, 2021 • edited Loading

Change Summary

Sample values

Release Target

Q/A

For mapping changes:

jonathan-buttner commented Sep 22, 2021 • edited Loading

ashokaditya commented Sep 23, 2021

jonathan-buttner left a comment

Choose a reason for hiding this comment

pzl left a comment

Choose a reason for hiding this comment

ashokaditya commented Sep 22, 2021 •

edited

Loading

jonathan-buttner commented Sep 22, 2021 •

edited

Loading