Skip to content

Commit

Permalink
Refactor schema reader to use intermediate schema structure (elastic#722
Browse files Browse the repository at this point in the history 
)

* refactor schema reader to use intermediate schema structure

* remove logic for '.' notation in reusable field names

* update changelog.next.md

* Add tests for intermediate schema data structures
  • Loading branch information
@marshallmain @dcode
marshallmain authored and dcode committed Apr 15, 2020
1 parent ed77806 commit 7700c88
Show file tree
Hide file tree
Showing 6 changed files with 624 additions and 416 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.next.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ Thanks, you're awesome :-) -->
#### Improvements

* ECS scripts now use Python 3.6+. #674
* schema_reader.py now reliably supports chaining reusable fieldsets together. #722

#### Deprecated

Expand Down
40 changes: 20 additions & 20 deletions generated/csv/fields.csv
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Example,Description
1.5.0-dev,true,client,client.address,keyword,extended,,Client network address.
1.5.0-dev,true,client,client.as.number,long,extended,15169,Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
1.5.0-dev,true,client,client.as.organization.name,keyword,extended,Google LLC,Organization name.
1.5.0-dev,true,client,as.organization.name.text,text,extended,Google LLC,Organization name.
1.5.0-dev,true,client,client.as.organization.name.text,text,extended,Google LLC,Organization name.
1.5.0-dev,true,client,client.bytes,long,core,184,Bytes sent from the client to the server.
1.5.0-dev,true,client,client.domain,keyword,core,,Client domain.
1.5.0-dev,true,client,client.geo.city_name,keyword,core,Montreal,City name.
Expand All @@ -36,14 +36,14 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Example,Description
1.5.0-dev,true,client,client.user.domain,keyword,extended,,Name of the directory the user is a member of.
1.5.0-dev,true,client,client.user.email,keyword,extended,,User email address.
1.5.0-dev,true,client,client.user.full_name,keyword,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,client,user.full_name.text,text,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,client,client.user.full_name.text,text,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,client,client.user.group.domain,keyword,extended,,Name of the directory the group is a member of.
1.5.0-dev,true,client,client.user.group.id,keyword,extended,,Unique identifier for the group on the system/platform.
1.5.0-dev,true,client,client.user.group.name,keyword,extended,,Name of the group.
1.5.0-dev,true,client,client.user.hash,keyword,extended,,Unique user hash to correlate information for a user in anonymized form.
1.5.0-dev,true,client,client.user.id,keyword,core,,One or multiple unique identifiers of the user.
1.5.0-dev,true,client,client.user.name,keyword,core,albert,Short name or login of the user.
1.5.0-dev,true,client,user.name.text,text,core,albert,Short name or login of the user.
1.5.0-dev,true,client,client.user.name.text,text,core,albert,Short name or login of the user.
1.5.0-dev,true,cloud,cloud.account.id,keyword,extended,666777888999,The cloud account or organization id.
1.5.0-dev,true,cloud,cloud.availability_zone,keyword,extended,us-east-1c,Availability zone in which this host is running.
1.5.0-dev,true,cloud,cloud.instance.id,keyword,extended,i-1234567890abcdef0,Instance ID of the host machine.
Expand All @@ -60,7 +60,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Example,Description
1.5.0-dev,true,destination,destination.address,keyword,extended,,Destination network address.
1.5.0-dev,true,destination,destination.as.number,long,extended,15169,Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
1.5.0-dev,true,destination,destination.as.organization.name,keyword,extended,Google LLC,Organization name.
1.5.0-dev,true,destination,as.organization.name.text,text,extended,Google LLC,Organization name.
1.5.0-dev,true,destination,destination.as.organization.name.text,text,extended,Google LLC,Organization name.
1.5.0-dev,true,destination,destination.bytes,long,core,184,Bytes sent from the destination to the source.
1.5.0-dev,true,destination,destination.domain,keyword,core,,Destination domain.
1.5.0-dev,true,destination,destination.geo.city_name,keyword,core,Montreal,City name.
Expand All @@ -82,14 +82,14 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Example,Description
1.5.0-dev,true,destination,destination.user.domain,keyword,extended,,Name of the directory the user is a member of.
1.5.0-dev,true,destination,destination.user.email,keyword,extended,,User email address.
1.5.0-dev,true,destination,destination.user.full_name,keyword,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,destination,user.full_name.text,text,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,destination,destination.user.full_name.text,text,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,destination,destination.user.group.domain,keyword,extended,,Name of the directory the group is a member of.
1.5.0-dev,true,destination,destination.user.group.id,keyword,extended,,Unique identifier for the group on the system/platform.
1.5.0-dev,true,destination,destination.user.group.name,keyword,extended,,Name of the group.
1.5.0-dev,true,destination,destination.user.hash,keyword,extended,,Unique user hash to correlate information for a user in anonymized form.
1.5.0-dev,true,destination,destination.user.id,keyword,core,,One or multiple unique identifiers of the user.
1.5.0-dev,true,destination,destination.user.name,keyword,core,albert,Short name or login of the user.
1.5.0-dev,true,destination,user.name.text,text,core,albert,Short name or login of the user.
1.5.0-dev,true,destination,destination.user.name.text,text,core,albert,Short name or login of the user.
1.5.0-dev,true,dns,dns.answers,object,extended,,Array of DNS answers.
1.5.0-dev,true,dns,dns.answers.class,keyword,extended,IN,The class of DNS data contained in this resource record.
1.5.0-dev,true,dns,dns.answers.data,keyword,extended,10.10.10.10,The data describing the resource.
Expand Down Expand Up @@ -195,25 +195,25 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Example,Description
1.5.0-dev,true,host,host.name,keyword,core,,Name of the host.
1.5.0-dev,true,host,host.os.family,keyword,extended,debian,"OS family (such as redhat, debian, freebsd, windows)."
1.5.0-dev,true,host,host.os.full,keyword,extended,Mac OS Mojave,"Operating system name, including the version or code name."
1.5.0-dev,true,host,os.full.text,text,extended,Mac OS Mojave,"Operating system name, including the version or code name."
1.5.0-dev,true,host,host.os.full.text,text,extended,Mac OS Mojave,"Operating system name, including the version or code name."
1.5.0-dev,true,host,host.os.kernel,keyword,extended,4.4.0-112-generic,Operating system kernel version as a raw string.
1.5.0-dev,true,host,host.os.name,keyword,extended,Mac OS X,"Operating system name, without the version."
1.5.0-dev,true,host,os.name.text,text,extended,Mac OS X,"Operating system name, without the version."
1.5.0-dev,true,host,host.os.name.text,text,extended,Mac OS X,"Operating system name, without the version."
1.5.0-dev,true,host,host.os.platform,keyword,extended,darwin,"Operating system platform (such centos, ubuntu, windows)."
1.5.0-dev,true,host,host.os.version,keyword,extended,10.14.1,Operating system version as a raw string.
1.5.0-dev,true,host,host.type,keyword,core,,Type of host.
1.5.0-dev,true,host,host.uptime,long,extended,1325,Seconds the host has been up.
1.5.0-dev,true,host,host.user.domain,keyword,extended,,Name of the directory the user is a member of.
1.5.0-dev,true,host,host.user.email,keyword,extended,,User email address.
1.5.0-dev,true,host,host.user.full_name,keyword,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,host,user.full_name.text,text,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,host,host.user.full_name.text,text,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,host,host.user.group.domain,keyword,extended,,Name of the directory the group is a member of.
1.5.0-dev,true,host,host.user.group.id,keyword,extended,,Unique identifier for the group on the system/platform.
1.5.0-dev,true,host,host.user.group.name,keyword,extended,,Name of the group.
1.5.0-dev,true,host,host.user.hash,keyword,extended,,Unique user hash to correlate information for a user in anonymized form.
1.5.0-dev,true,host,host.user.id,keyword,core,,One or multiple unique identifiers of the user.
1.5.0-dev,true,host,host.user.name,keyword,core,albert,Short name or login of the user.
1.5.0-dev,true,host,user.name.text,text,core,albert,Short name or login of the user.
1.5.0-dev,true,host,host.user.name.text,text,core,albert,Short name or login of the user.
1.5.0-dev,true,http,http.request.body.bytes,long,extended,887,Size in bytes of the request body.
1.5.0-dev,true,http,http.request.body.content,keyword,extended,Hello world,The full HTTP request body.
1.5.0-dev,true,http,http.request.body.content.text,text,extended,Hello world,The full HTTP request body.
Expand Down Expand Up @@ -263,10 +263,10 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Example,Description
1.5.0-dev,true,observer,observer.name,keyword,extended,1_proxySG,Custom name of the observer.
1.5.0-dev,true,observer,observer.os.family,keyword,extended,debian,"OS family (such as redhat, debian, freebsd, windows)."
1.5.0-dev,true,observer,observer.os.full,keyword,extended,Mac OS Mojave,"Operating system name, including the version or code name."
1.5.0-dev,true,observer,os.full.text,text,extended,Mac OS Mojave,"Operating system name, including the version or code name."
1.5.0-dev,true,observer,observer.os.full.text,text,extended,Mac OS Mojave,"Operating system name, including the version or code name."
1.5.0-dev,true,observer,observer.os.kernel,keyword,extended,4.4.0-112-generic,Operating system kernel version as a raw string.
1.5.0-dev,true,observer,observer.os.name,keyword,extended,Mac OS X,"Operating system name, without the version."
1.5.0-dev,true,observer,os.name.text,text,extended,Mac OS X,"Operating system name, without the version."
1.5.0-dev,true,observer,observer.os.name.text,text,extended,Mac OS X,"Operating system name, without the version."
1.5.0-dev,true,observer,observer.os.platform,keyword,extended,darwin,"Operating system platform (such centos, ubuntu, windows)."
1.5.0-dev,true,observer,observer.os.version,keyword,extended,10.14.1,Operating system version as a raw string.
1.5.0-dev,true,observer,observer.product,keyword,extended,s200,The product name of the observer.
Expand Down Expand Up @@ -363,7 +363,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Example,Description
1.5.0-dev,true,server,server.address,keyword,extended,,Server network address.
1.5.0-dev,true,server,server.as.number,long,extended,15169,Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
1.5.0-dev,true,server,server.as.organization.name,keyword,extended,Google LLC,Organization name.
1.5.0-dev,true,server,as.organization.name.text,text,extended,Google LLC,Organization name.
1.5.0-dev,true,server,server.as.organization.name.text,text,extended,Google LLC,Organization name.
1.5.0-dev,true,server,server.bytes,long,core,184,Bytes sent from the server to the client.
1.5.0-dev,true,server,server.domain,keyword,core,,Server domain.
1.5.0-dev,true,server,server.geo.city_name,keyword,core,Montreal,City name.
Expand All @@ -385,14 +385,14 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Example,Description
1.5.0-dev,true,server,server.user.domain,keyword,extended,,Name of the directory the user is a member of.
1.5.0-dev,true,server,server.user.email,keyword,extended,,User email address.
1.5.0-dev,true,server,server.user.full_name,keyword,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,server,user.full_name.text,text,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,server,server.user.full_name.text,text,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,server,server.user.group.domain,keyword,extended,,Name of the directory the group is a member of.
1.5.0-dev,true,server,server.user.group.id,keyword,extended,,Unique identifier for the group on the system/platform.
1.5.0-dev,true,server,server.user.group.name,keyword,extended,,Name of the group.
1.5.0-dev,true,server,server.user.hash,keyword,extended,,Unique user hash to correlate information for a user in anonymized form.
1.5.0-dev,true,server,server.user.id,keyword,core,,One or multiple unique identifiers of the user.
1.5.0-dev,true,server,server.user.name,keyword,core,albert,Short name or login of the user.
1.5.0-dev,true,server,user.name.text,text,core,albert,Short name or login of the user.
1.5.0-dev,true,server,server.user.name.text,text,core,albert,Short name or login of the user.
1.5.0-dev,true,service,service.ephemeral_id,keyword,extended,8a4f500f,Ephemeral identifier of this service.
1.5.0-dev,true,service,service.id,keyword,core,d37e5ebfe0ae6c4972dbe9f0174a1637bb8247f6,Unique identifier of the running service.
1.5.0-dev,true,service,service.name,keyword,core,elasticsearch-metrics,Name of the service.
Expand All @@ -403,7 +403,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Example,Description
1.5.0-dev,true,source,source.address,keyword,extended,,Source network address.
1.5.0-dev,true,source,source.as.number,long,extended,15169,Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
1.5.0-dev,true,source,source.as.organization.name,keyword,extended,Google LLC,Organization name.
1.5.0-dev,true,source,as.organization.name.text,text,extended,Google LLC,Organization name.
1.5.0-dev,true,source,source.as.organization.name.text,text,extended,Google LLC,Organization name.
1.5.0-dev,true,source,source.bytes,long,core,184,Bytes sent from the source to the destination.
1.5.0-dev,true,source,source.domain,keyword,core,,Source domain.
1.5.0-dev,true,source,source.geo.city_name,keyword,core,Montreal,City name.
Expand All @@ -425,14 +425,14 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Example,Description
1.5.0-dev,true,source,source.user.domain,keyword,extended,,Name of the directory the user is a member of.
1.5.0-dev,true,source,source.user.email,keyword,extended,,User email address.
1.5.0-dev,true,source,source.user.full_name,keyword,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,source,user.full_name.text,text,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,source,source.user.full_name.text,text,extended,Albert Einstein,"User's full name, if available."
1.5.0-dev,true,source,source.user.group.domain,keyword,extended,,Name of the directory the group is a member of.
1.5.0-dev,true,source,source.user.group.id,keyword,extended,,Unique identifier for the group on the system/platform.
1.5.0-dev,true,source,source.user.group.name,keyword,extended,,Name of the group.
1.5.0-dev,true,source,source.user.hash,keyword,extended,,Unique user hash to correlate information for a user in anonymized form.
1.5.0-dev,true,source,source.user.id,keyword,core,,One or multiple unique identifiers of the user.
1.5.0-dev,true,source,source.user.name,keyword,core,albert,Short name or login of the user.
1.5.0-dev,true,source,user.name.text,text,core,albert,Short name or login of the user.
1.5.0-dev,true,source,source.user.name.text,text,core,albert,Short name or login of the user.
1.5.0-dev,true,threat,threat.framework,keyword,extended,MITRE ATT&CK,Threat classification framework.
1.5.0-dev,true,threat,threat.tactic.id,keyword,extended,TA0040,Threat tactic id.
1.5.0-dev,true,threat,threat.tactic.name,keyword,extended,impact,Threat tactic.
Expand Down Expand Up @@ -504,10 +504,10 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Example,Description
1.5.0-dev,true,user_agent,user_agent.original.text,text,extended,"Mozilla/5.0 (iPhone; CPU iPhone OS 12_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1",Unparsed user_agent string.
1.5.0-dev,true,user_agent,user_agent.os.family,keyword,extended,debian,"OS family (such as redhat, debian, freebsd, windows)."
1.5.0-dev,true,user_agent,user_agent.os.full,keyword,extended,Mac OS Mojave,"Operating system name, including the version or code name."
1.5.0-dev,true,user_agent,os.full.text,text,extended,Mac OS Mojave,"Operating system name, including the version or code name."
1.5.0-dev,true,user_agent,user_agent.os.full.text,text,extended,Mac OS Mojave,"Operating system name, including the version or code name."
1.5.0-dev,true,user_agent,user_agent.os.kernel,keyword,extended,4.4.0-112-generic,Operating system kernel version as a raw string.
1.5.0-dev,true,user_agent,user_agent.os.name,keyword,extended,Mac OS X,"Operating system name, without the version."
1.5.0-dev,true,user_agent,os.name.text,text,extended,Mac OS X,"Operating system name, without the version."
1.5.0-dev,true,user_agent,user_agent.os.name.text,text,extended,Mac OS X,"Operating system name, without the version."
1.5.0-dev,true,user_agent,user_agent.os.platform,keyword,extended,darwin,"Operating system platform (such centos, ubuntu, windows)."
1.5.0-dev,true,user_agent,user_agent.os.version,keyword,extended,10.14.1,Operating system version as a raw string.
1.5.0-dev,true,user_agent,user_agent.version,keyword,extended,12.0,Version of the user agent.
Expand Down

0 comments on commit 7700c88

Please sign in to comment.