Elastic Common Schema
Switch branches/tags
Nothing to show
Clone or download
ruflin and andrewkroh Add Beats compatible fields.yml for ECS (#108)
* Add Beats compatible fields.yml for ECS

Add `type: group` to each prefix.

* Indent and add fields to make check

* add field generation to generate target
Latest commit 3ca1f4b Sep 3, 2018

README.md

WARNING: THIS IS WORK IN PROGRESS

Elastic Common Schema (ECS)

The Elastic Common Schema (ECS) defines a common set of fields for ingesting data into Elasticsearch. A common schema helps you correlate data from sources like logs and metrics or IT operations analytics and security analytics.

ECS is still under development and backward compatibility is not guaranteed. Any feedback on the general structure, missing fields, or existing fields is appreciated. For contributions please read the Contributing Guide.

The current version of ECS is 0.1.0.

In this readme

Fields

ECS defines these fields.

Base fields

The base set contains all fields which are on the top level. These fields are common across all types of events.

Field Description Type Multi Field Example
@timestamp Date/time when the event originated.
For log events this is the date/time when the event was generated, and not when it was read.
Required field for all events.
date 2016-05-23T08:05:34.853Z
tags List of keywords used to tag each event. keyword ["production", "env2"]
labels Key/value pairs.
Can be used to add meta information to events. Should not contain nested objects. All values are stored as keyword.
Example: docker and k8s labels.
object {key1: value1, key2: value2}
message For log events the message field contains the log message.
In other use cases the message field can be used to concatenate different values which are then freely searchable. If multiple messages exist, they can be combined into one message.
text Hello World

Agent fields

The agent fields contain the data about the agent/client/shipper that created the event.

Field Description Type Multi Field Example
agent.version Version of the agent. keyword 6.0.0-rc2
agent.name Name of the agent. keyword filebeat
agent.id Unique identifier of this agent (if one exists).
Example: For Beats this would be beat.id.
keyword 8a4f500d
agent.ephemeral_id Ephemeral identifier of this agent (if one exists).
This id normally changes across restarts, but agent.id does not.
keyword 8a4f500f

Examples: In the case of Beats for logs, the agent.name is filebeat. For APM, it is the agent running in the app/service. The agent information does not change if data is sent through queuing systems like Kafka, Redis, or processing systems such as Logstash or APM Server.

Cloud fields

Fields related to the cloud or infrastructure the events are coming from.

Field Description Type Multi Field Example
cloud.provider Name of the cloud provider. Example values are ec2, gce, or digitalocean. keyword ec2
cloud.availability_zone Availability zone in which this host is running. keyword us-east-1c
cloud.region Region in which this host is running. keyword us-east-1
cloud.instance.id Instance ID of the host machine. keyword i-1234567890abcdef0
cloud.instance.name Instance name of the host machine. keyword
cloud.machine.type Machine type of the host machine. keyword t2.medium
cloud.account.id The cloud account or organization id used to identify different entities in a multi-tenant environment.
Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
keyword 666777888999

Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.

Container fields

Container fields are used for meta information about the specific container that is the source of information. These fields help correlate data based containers from any runtime.

Field Description Type Multi Field Example
container.runtime Runtime managing this container. keyword docker
container.id Unique container id. keyword
container.image.name Name of the image the container was built on. keyword
container.image.tag Container image tag. keyword
container.name Container name. keyword
container.labels Image labels. object

Destination fields

Destination fields describe details about the destination of a packet/event.

Field Description Type Multi Field Example
destination.ip IP address of the destination.
Can be one or multiple IPv4 or IPv6 addresses.
ip
destination.hostname Hostname of the destination. keyword
destination.port Port of the destination. long
destination.mac MAC address of the destination. keyword
destination.domain Destination domain. keyword
destination.subdomain Destination subdomain. keyword

Device fields

Device fields are used to provide additional information about the device that is the source of the information. This could be a firewall, network device, etc.

Field Description Type Multi Field Example
device.mac MAC address of the device keyword
device.ip IP address of the device. ip
device.hostname Hostname of the device. keyword
device.vendor Device vendor information. text
device.version Device version. keyword
device.serial_number Device serial number. keyword
device.timezone.offset.sec Timezone offset of the host in seconds.
Number of seconds relative to UTC. If the offset is -01:30 the value will be -5400.
long -5400
device.type The type of the device the data is coming from.
There is no predefined list of device types. Some examples are endpoint, firewall, ids, ips, proxy.
keyword firewall

Error fields

These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.

Field Description Type Multi Field Example
error.id Unique identifier for the error. keyword
error.message Error message. text
error.code Error code describing the error. keyword

Event fields

The event fields are used for context information about the data itself.

Field Description Type Multi Field Example
event.id Unique ID to describe the event. keyword 8a4f500d
event.category Event category.
This can be a user defined category.
keyword metrics
event.type A type given to this kind of event which can be used for grouping.
This is normally defined by the user.
keyword nginx-stats-metrics
event.action The action captured by the event. The type of action will vary from system to system but is likely to include actions by security services, such as blocking or quarantining; as well as more generic actions such as login events, file i/o or proxy forwarding events.
The value is normally defined by the user.
keyword reject
event.module Name of the module this data is coming from.
This information is coming from the modules used in Beats or Logstash.
keyword mysql
event.dataset Name of the dataset.
The concept of a dataset (fileset / metricset) is used in Beats as a subset of modules. It contains the information which is currently stored in metricset.name and metricset.module or fileset.name.
keyword stats
event.severity Severity describes the severity of the event. What the different severity values mean can very different between use cases. It's up to the implementer to make sure severities are consistent across events. long 7
event.original Raw text message of entire event. Used to demonstrate log integrity.
This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from _source.
keyword Sep 19 08:26:10 host CEF:0|Security| threatmanager|1.0|100| worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2spt=1232
event.hash Hash (perhaps logstash fingerprint) of raw field to be able to demonstrate log integrity. keyword 123456789012345678901234567890ABCD
event.version The version field contains the version an event for ECS adheres to.
This field should be provided as part of each event to make it possible to detect to which ECS version an event belongs.
event.version is a required field and must exist in all events. It describes which ECS version the event adheres to.
The current version is 0.1.0.
keyword 0.1.0
event.duration Duration of the event in nanoseconds. long
event.created event.created contains the date when the event was created.
This timestamp is distinct from @timestamp in that @timestamp contains the processed timestamp. For logs these two timestamps can be different as the timestamp in the log line and when the event is read for example by Filebeat are not identical. @timestamp must contain the timestamp extracted from the log line, event.created when the log line is read. The same could apply to package capturing where @timestamp contains the timestamp extracted from the network package and event.created when the event was created.
In case the two timestamps are identical, @timestamp should be used.
date
event.risk_score Risk score or priority of the event (e.g. security solutions). Use your system's original value here. float
event.risk_score_norm Normalized risk score or priority of the event, on a scale of 0 to 100.
This is mainly useful if you use more than one system that assigns risk scores, and you want to see a normalized value across all systems.
float

File fields

File fields provide details about each file.

Field Description Type Multi Field Example
file.path Path to the file. text
file.path.raw Path to the file. This is a non-analyzed field that is useful for aggregations. keyword 1
file.target_path Target path for symlinks. text
file.target_path.raw Path to the file. This is a non-analyzed field that is useful for aggregations. keyword 1
file.extension File extension.
This should allow easy filtering by file extensions.
keyword png
file.type File type (file, dir, or symlink). keyword
file.device Device that is the source of the file. keyword
file.inode Inode representing the file in the filesystem. keyword
file.uid The user ID (UID) or security identifier (SID) of the file owner. keyword
file.owner File owner's username. keyword
file.gid Primary group ID (GID) of the file. keyword
file.group Primary group name of the file. keyword
file.mode Mode of the file in octal representation. keyword 416
file.size File size in bytes (field is only added when type is file). long
file.mtime Last time file content was modified. date
file.ctime Last time file metadata changed. date

Geo fields

Geo fields can carry data about a specific location related to an event or geo information for an IP field.

Field Description Type Multi Field Example
geo.continent_name Name of the continent. keyword
geo.country_iso_code Country ISO code. keyword
geo.location Longitude and latitude. geo_point
geo.region_name Region name. keyword
geo.city_name City name. keyword

Host fields

Host fields provide information related to a host. A host can be a physical machine, a virtual machine, or a Docker container.

Normally the host information is related to the machine on which the event was generated/collected, but they can be used differently if needed.

Field Description Type Multi Field Example
host.timezone.offset.sec Timezone offset of the host in seconds.
Number of seconds relative to UTC. If the offset is -01:30 the value will be -5400.
long -5400
host.name host.name is the hostname of the host.
It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.
keyword
host.id Unique host id.
As hostname is not always unique, use values that are meaningful in your environment.
Example: The current usage of beat.name.
keyword
host.ip Host ip address. ip
host.mac Host mac address. keyword
host.type Type of host.
For Cloud providers this can be the machine type like t2.medium. If vm, this could be the container, for example, or other information meaningful in your environment.
keyword
host.os.platform Operating system platform (centos, ubuntu, windows, etc.) keyword darwin
host.os.name Operating system name. keyword Mac OS X
host.os.family OS family (redhat, debian, freebsd, windows, etc.) keyword debian
host.os.version Operating system version. keyword 10.12.6
host.architecture Operating system architecture. keyword x86_64

HTTP fields

Fields related to HTTP requests and responses.

Field Description Type Multi Field Example
http.request.method Http request method. keyword GET, POST, PUT
http.response.status_code Http response status code. long 404
http.response.body The full http response body. text Hello world
http.version Http version. keyword 1.1

Kubernetes fields

Kubernetes fields are used for Kubernetes meta information. This information helps correlate data from Kubernetes setups.

Field Description Type Multi Field Example
kubernetes.pod.name Kubernetes pod name keyword
kubernetes.namespace Kubernetes namespace keyword
kubernetes.labels Kubernetes labels map object
kubernetes.annotations Kubernetes annotations map object
kubernetes.container.name Kubernetes container name. This name is unique within the pod only. It is different from the underlying container.name field. keyword

Log fields

Fields which are specific to log events.

Field Description Type Multi Field Example
log.level Log level of the log event.
Some examples are WARN, ERR, INFO.
keyword ERR
log.line Line number the log event was collected from. long 18
log.offset Offset of the beginning of the log event. long 12
log.original This is the original log message and contains the full log message before splitting it up in multiple parts.
In contrast to the message field which can contain an extracted part of the log message, this field contains the original, full log message. It can have already some modifications applied like encoding or new lines removed to clean up the log message.
This field is not indexed and doc_values are disabled so it can't be queried but the value can be retrieved from _source.
keyword Sep 19 08:26:10 localhost My log

Network fields

Fields related to network data.

Field Description Type Multi Field Example
network.name Name given by operators to sections of their network. text Guest Wifi
network.name.raw Name given by operators to sections of their network. keyword 1
network.protocol Network protocol name. keyword http
network.direction Direction of the network traffic.
Recommended values are:
* inbound
* outbound
* unknown
keyword inbound
network.forwarded_ip Host IP address when the source IP address is the proxy. ip 192.1.1.2
network.inbound.bytes Network inbound bytes. long 184
network.inbound.packets Network inbound packets. long 12
network.outbound.bytes Network outbound bytes. long 184
network.outbound.packets Network outbound packets. long 12
network.total.bytes Network total bytes. The sum of inbound.bytes + outbound.bytes. long 368
network.total.packets Network outbound packets. The sum of inbound.packets + outbound.packets long 24

Organization fields

The organization fields enrich data with information about the company or entity the data is associated with. These fields help you arrange or filter data stored in an index by one or multiple organizations.

Field Description Type Multi Field Example
organization.name Organization name. text
organization.id Unique identifier for the organization. keyword

Operating System fields

The OS fields contain information about the operating system. These fields are often used inside other prefixes, such as host.os.* or user_agent.os.*.

Field Description Type Multi Field Example
os.platform Operating system platform (such centos, ubuntu, windows). keyword darwin
os.name Operating system name. keyword Mac OS X
os.family OS family (such as redhat, debian, freebsd, windows). keyword debian
os.version Operating system version as a raw string. keyword 10.12.6-rc2
os.kernel Operating system kernel version as a raw string. keyword 4.4.0-112-generic

Process fields

These fields contain information about a process. These fields can help you correlate metrics information with a process id/name from a log message. The process.pid often stays in the metric itself and is copied to the global field for correlation.

Field Description Type Multi Field Example
process.args Process arguments.
May be filtered to protect sensitive information.
keyword ['-l', 'user', '10.0.0.16']
process.name Process name.
Sometimes called program name or similar.
keyword ssh
process.pid Process id. long
process.ppid Process parent id. long
process.title Process title.
The proctitle, often the same as process name.
keyword

Service fields

The service fields describe the service for or from which the data was collected. These fields help you find and correlate logs for a specific service and version.

Field Description Type Multi Field Example
service.id Unique identifier of the running service.
This id should uniquely identify this service. This makes it possible to correlate logs and metrics for one specific service.
Example: If you are experiencing issues with one redis instance, you can filter on that id to see metrics and logs for that single instance.
keyword d37e5ebfe0ae6c4972dbe9f0174a1637bb8247f6
service.name Name of the service data is collected from.
The name can be used to group and correlate logs and metrics from one service.
Example: If logs or metrics are collected from Redis, service.name would be redis.
keyword elasticsearch
service.type Service type. keyword
service.state Current state of the service. keyword
service.version Version of the service the data was collected from.
This allows to look at a data set only for a specific version of a service.
keyword 3.2.4
service.ephemeral_id Ephemeral identifier of this service (if one exists).
This id normally changes across restarts, but service.id does not.
keyword 8a4f500f

Source fields

Source fields describe details about the source of the event.

Field Description Type Multi Field Example
source.ip IP address of the source.
Can be one or multiple IPv4 or IPv6 addresses.
ip
source.hostname Hostname of the source. keyword
source.port Port of the source. long
source.mac MAC address of the source. keyword
source.domain Source domain. keyword
source.subdomain Source subdomain. keyword

TLS fields

The tls fields contain the TLS related data about a specific connection.

Field Description Type Multi Field Example
tls.version TLS version. keyword TLSv1.2
tls.certificates An array of certificates. keyword
tls.servername Server name requested by the client. keyword localhost
tls.ciphersuite Name of the cipher used for the communication. keyword ECDHE-ECDSA-AES-128-CBC-SHA

As an example in the case of Filebeat and the TCP input, the version field would be the version of the TLS protocol in use, the certificates would be the chain of certificates provided by the client and the ciphersuite is the encryption algorithm used for the communication.

URL fields

URL fields provide a complete URL, with scheme, host, and path. The URL object can be reused in other prefixes, such as host.url.* for example. Keep the structure consistent whenever you use URL fields.

Field Description Type Multi Field Example
url.href Full url. The field is stored as keyword.
url.href is a [multi field](https://www.elastic.co/guide/en/ elasticsearch/reference/6.2/ multi-fields.html#_multi_fields_with_multiple_analyzers). The data is stored as keyword url.href and test url.href.analyzed. These fields enable you to run a query against part of the url still works splitting up the URL at ingest time.
href is an analyzed field so the parsed information can be accessed through href.analyzed in queries.
text https://elastic.co:443/search?q=elasticsearch#top
url.href.raw The full URL. This is a non-analyzed field that is useful for aggregations. keyword 1
url.scheme Scheme of the request, such as "https".
Note: The : is not part of the scheme.
keyword https
url.host.name Hostname of the request, such as "example.com".
For correlation the this field can be copied into the host.name field.
keyword elastic.co
url.port Port of the request, such as 443. integer 443
url.path Path of the request, such as "/search". text
url.path.raw URL path. A non-analyzed field that is useful for aggregations. keyword 1
url.query The query field describes the query string of the request, such as "q=elasticsearch".
The ? is excluded from the query string. If a URL contains no ?, there is no query field. If there is a ? but no query, the query field exists with an empty string. The exists query can be used to differentiate between the two cases.
text
url.query.raw URL query part. A non-analyzed field that is useful for aggregations. keyword 1
url.fragment Portion of the url after the #, such as "top".
The # is not part of the fragment.
keyword
url.username Username of the request. keyword
url.password Password of the request. keyword

User fields

The user fields describe information about the user that is relevant to the event. Fields can have one entry or multiple entries. If a user has more than one id, provide an array that includes all of them.

Field Description Type Multi Field Example
user.id One or multiple unique identifiers of the user. keyword
user.name Name of the user.
The field is a keyword, and will not be tokenized.
keyword
user.email User email address. keyword
user.hash Unique user hash to correlate information for a user in anonymized form.
Useful if user.id or user.name contain confidential information and cannot be used.
keyword

User agent fields

The user_agent fields normally come from a browser request. They often show up in web service logs coming from the parsed user agent string.

Field Description Type Multi Field Example
user_agent.original Unparsed version of the user_agent. text
user_agent.device Name of the physical device. keyword
user_agent.version Version of the physical device. keyword
user_agent.major Major version of the user agent. long
user_agent.minor Minor version of the user agent. long
user_agent.patch Patch version of the user agent. keyword
user_agent.name Name of the user agent. keyword Chrome
user_agent.os.name Name of the operating system. keyword
user_agent.os.version Version of the operating system. keyword
user_agent.os.major Major version of the operating system. long
user_agent.os.minor Minor version of the operating system. long

Use cases

These are example on how ECS fields can be used in different use cases. Most use cases not only contain ECS fields but additional fields which are not in ECS to describe the full use case. The fields which are not in ECS are in italic.

Contributions of additional uses cases on top of ECS are welcome.

Implementing ECS

Guidelines

  • The document MUST have the @timestamp field.
  • The data type defined for an ECS field MUST be used.
  • It SHOULD have the field event.version to define which version of ECS it uses.
  • As many fields as possible should be mapped to ECS.

Writing fields

  • All fields must be lower case
  • Combine words using underscore
  • No special characters except _

Naming fields

  • Present tense. Use present tense unless field describes historical information.
  • Singular or plural. Use singular and plural names properly to reflect the field content. For example, use requests_per_sec rather than request_per_sec.
  • General to specific. Organise the prefixes from general to specific to allow grouping fields into objects with a prefix like host.*.
  • Avoid repetition. Avoid stuttering of words. If part of the field name is already in the prefix, do not repeat it. Example: host.host_ip should be host.ip.
  • Use prefixes. Fields must be prefixed except for the base fields. For example all host fields are prefixed with host.. See dot notation in FAQ for more details.
  • Do not use abbreviations. (A few exceptions like ip exist.)

Understanding ECS conventions

Multi-fields text indexing

ElasticSearch can index text multiple ways:

  • text indexing allows for full text search, or searching arbitrary words that are part of the field.
  • keyword indexing allows for much faster exact match and prefix search, and allows for aggregations (what Kibana visualizations are built on).

In some cases, only one type of indexing makes sense for a field.

However there are cases where both types of indexing can be useful, and we want to index both ways. As an example, log messages can sometimes be short enough that it makes sense to sort them by frequency (that's an aggregation). They can also be long and varied enough that full text search can be useful on them.

Whenever both types of indexing are helpful, we use multi-fields indexing. The convention used is the following:

  • foo: text indexing. The top level of the field (its plain name) is used for full text search.
  • foo.raw: keyword indexing. The nested field has suffix .raw and is what you will use for aggregations.
    • Performance tip: when filtering your stream in Kibana (or elsewhere), if you are filtering for an exact match or doing a prefix search, both text and keyword field can be used, but doing so on the keyword field (named .raw) will be much faster and less memory intensive.

Keyword only fields

The fields that only make sense as type keyword are not named foo.raw, the plain field (foo) will be of type keyword, with no nested field.

IDs are keywords not integers

Despite the fact that IDs are often integers in various systems, this is not always the case. Since we want to make it possible to map as many data sources to ECS as possible, we default to using the keyword type for IDs.

FAQ

What are the benefits of using ECS?

The benefits to a user adopting these fields and names in their clusters are:

  • Data correlation. Ability to easily correlate data from the same or different sources, including:
    • data from metrics, logs, and apm
    • data from the same machines/hosts
    • data from the same service
  • Ease of recall. Improved ability to remember commonly used field names (because there is a single set, not a set per data source)
  • Ease of deduction. Improved ability to deduce field names (because the field naming follows a small number of rules with few exceptions)
  • Reuse. Ability to re-use analysis content (searches, visualizations, dashboards, alerts, reports, and ML jobs) across multiple data sources
  • Future proofing. Ability to use any future Elastic-provided analysis content in your environment without modifications

What if I have fields that conflict with ECS?

The rename processor can help you resolve field conflicts. For example, imagine that you already have a field called "user," but ECS employs user as an object. You can use the rename processor on ingest time to rename your field to the matching ECS field. If your field does not match ECS, you can rename your field to user.value instead.

What if my events have additional fields?

Events may contain fields in addition to ECS fields. These fields can follow the ECS naming and writing rules, but this is not a requirement.

Why does ECS use a dot notation instead of an underline notation?

There are two common key formats for ingesting data into Elasticsearch:

  • Dot notation: user.firstname: Nicolas, user.lastname: Ruflin
  • Underline notation: user_firstname: Nicolas, user_lastname: Ruflin

For ECS we decided to use the dot notation. Here's some background on this decision.

What is the difference between the two notations?

Ingesting user.firstname: Nicolas and user.lastname: Ruflin is identical to ingesting the following JSON:

"user": {
  "firstname": "Nicolas",
  "lastname": "Ruflin"
}

In Elasticsearch, user is represented as an object datatype. In the case of the underline notation, both are just string datatypes.

NOTE: ECS does not use nested datatypes, which are arrays of objects.

Advantages of dot notation

With dot notation, each prefix in Elasticsearch is an object. Each object can have parameters that control how fields inside the object are treated. In the context of ECS, for example, these parameters would allow you to disable dynamic property creation for certain prefixes.

Individual objects give you more flexibility on both the ingest and the event sides. In Elasticsearch, for example, you can use the remove processor to drop complete objects instead of selecting each key inside. You don't have to know ahead of time which keys will be in an object.

In Beats, you can simplify the creation of events. For example, you can treat each object as an object (or struct in Golang), which makes constructing and modifying each part of the final event easier.

Disadvantage of dot notation

In Elasticsearch, each key can only have one type. For example, if user is an object, you can't use it as akeyword type in the same index, like {"user": "nicolas ruflin"}. This restriction can be an issue in certain datasets. For the ECS data itself, this is not an issue because all fields are predefined.

What if I already use the underline notation?

Mixing the underline notation with the ECS dot notation is not a problem. As long as there are no conflicts, they can coexist in the same document.