Permalink
Browse files

Parsers done

  • Loading branch information...
thomashumio committed Jun 20, 2018
1 parent a206f9f commit 05f15d06ed3ac6deeefeb6c3cd0113698e9eedbe
@@ -16,9 +16,9 @@ In fact, the UI is constructed using solely the public API.

## REST and GraphQL

Humio has a mixture of GraphQL and REST endpoint. You can read about the
[motivation of using GraphQL]({{< relref "graphql.md" >}}),
as well as an introduction to GraphQL if you are not familiar.
Humio has a mixture of GraphQL and REST endpoints. You can read about the
[motivation behind using GraphQL]({{< relref "graphql.md" >}}),
as well as an introduction to GraphQL if you are not familiar with the technology.

{{% notice info %}}
We are in the process of migrating most of our API to GraphQL, but there are
@@ -15,7 +15,7 @@ typically use the endpoint where a parser is specified.
### Ingest data using a parser {#parser}

This API should be used, when a parser should be applied to the data.
It is possible to create [parsers]({{< relref "parsing.md" >}}) in Humio
It is possible to create [parsers]({{< ref "parsers/_index.md" >}}) in Humio

{{% notice note %}}
***Filebeat is another option for sending data that needs a parser***
@@ -50,7 +50,7 @@ Example sending 4 accesslog lines to Humio
```

The above example sends 4 accesslog lines to Humio. the parser is specified using the `type` field and is set to `accesslog`.
The parser accesslog should be specified in the repository. See [parsing]({{< relref "parsing.md" >}}) for details.
The parser accesslog should be specified in the repository. See [parsing]({{< relref "parsers/_index.md" >}}) for details.
The `fields` section is used to specify fields that should be added to each of the events when they are parsed. In the example all the accesslog events will get a host field telling the events came from webhost1.
It is possible to send events of different types in the same request. That is done by adding a new element to the outer array in the example above.
Tags can be specified in the parser pointed to by the `type` field
@@ -62,7 +62,7 @@ When sending events, you can set the following standard fields:
Name | Required | Description
------------ | ------------- |------------
`messages` | yes | The raw strings representing the events. Each string will be parsed by the parser specified by `type`.
`type` | yes | The [parser]({{< relref "parsing.md" >}}) Humio will use to parse the `messages`
`type` | yes | The [parser]({{< relref "parsers/_index.md" >}}) Humio will use to parse the `messages`
`fields` | no | Annotate each of the `messages` with these key-values. Values must be strings.
`tags` | no | Annotate each of the `messages` with these key-values as Tags. Please see other documentation on tags before using.

@@ -1,3 +1,5 @@
---
title: "Appendix"
---

{{% children %}}
@@ -24,7 +24,7 @@ You set `@type` to the name of one of Humio's built-in parsers or one of your ow

For now let os assume we can use the build-in [`kv`]({{< ref "kv.md" >}}) (Key-Value) parser, it extracts fields
from the log lines, anything of the form `key=value`.
See [Parsing Logs]({{< relref "parsing.md" >}}) for more information on parsing log data.
See [Parsing Logs]({{< relref "parsers/_index.md" >}}) for more information on parsing log data.

Example Filebeat configuration with a custom log type:

@@ -44,7 +44,7 @@ output.elasticsearch:
* `$REPOSITORY_NAME` - name of your repository on your server (e.g. `sandbox`)
* `$INGEST_TOKEN` - [ingest token]({{< relref "ingest_tokens.md" >}}) for your repository, (e.g. a string such as `fS6Kdlb0clqe0UwPcc4slvNFP3Qn1COzG9DEVLw7v0Ii`).
* `PATH_TO_YOUR_APPLICATION_LOG` - the file path to the log file you want to send.
* `PARSER_NAME` - the name of either one of the built-in parsers such as `kv` (Key-Value) or a [custom parser]({{< relref "parsing.md" >}}).
* `PARSER_NAME` - the name of either one of the built-in parsers such as `kv` (Key-Value) or a [custom parser]({{< relref "parsers/_index.md" >}}).

See the detailed documentation for [filebeat]({{< relref "filebeat.md" >}})

@@ -80,7 +80,7 @@ output.elasticsearch:
Notice the type is `syslog-utc`, which points to the built in syslog parser, expecting the timestamp to be in UTC time.
Often syslog timestamps are in local time. Go ahead and create a new parser with another timezone in Humio if necessary.
You can copy the built in syslog-utc and just change the timezone.
See [Parsing]({{< relref "parsing.md" >}}) for details.
See [Parsing]({{< relref "parsers/_index.md" >}}) for details.


Check out the [Filebeat]({{< relref "filebeat.md" >}}) page for more
@@ -3,13 +3,14 @@ title: "Setting Up Authentication"
weight: 2
---

{{% notice tip %}}
If you are just experimenting and playing around with Humio, you can probably
skip this page. **For production deployments you want to set up authentication, the first thing you do.**
{{% /notice %}}

If not configured otherwise Humio runs in `NO_AUTH` mode, meaning that there
are no access restrictions at all – anyone with access to the system, can do
anything.

If you are just experimenting and playing around with Humio, you can probably
just leave it as it is.

For any real deployment you want to change this the first thing you do.

Refer to the [Authentication Configuration]({{< ref "configuration/authentication/_index.md" >}}) for different login options.
Refer to the [Authentication Configuration]({{< ref "configuration/authentication/_index.md" >}}) for different login options.
@@ -113,7 +113,7 @@ To test it can be run like `/usr/share/filebeat/bin/filebeat -c /etc/filebeat/fi

Humio uses parsers to parse the data from Filebeat into events.
Parsers can extract fields from the text strings an add structure to the events.
For more information on parsers, see [parsing]({{< relref "parsing.md" >}}).
For more information on parsers, see [parsing]({{< relref "parsers/_index.md" >}}).

{{% notice note %}}
Take a look at Humio's [built-in parsers](/sending-data/parsers/built_in_parsers/).
@@ -128,7 +128,7 @@ For example, when sending a web server access log file to Humio, you can use the

### Parsing JSON data

Humio supports [JSON parsers]({{< relref "parsing.md#json-parser" >}}).
Humio supports [JSON parsers]({{< ref "json-parsers.md" >}}).
Filebeat processes logs line by line, so JSON parsing will only work if there is one JSON object per line.
Customize a JSON parser in Humio (do not use the JSON parsing built into Filebeat).

@@ -151,7 +151,7 @@ Humio saves data in Data Sources. You can provide a set of Tags to specify which
See [glossary](/glossary/#tags) for more information about tags and Data Sources.
The `type` configured in Filebeat is always used as tag. Other fields can be used
as tags as well by defining the fields as `tagFields` in the
[parser]({{< relref "parsing.md" >}}) pointed to by the `type`.
[parser]({{< relref "parsers/_index.md" >}}) pointed to by the `type`.
In Humio tags always start with a #. When turning a field into a tag it will
be prepended with `#`.

@@ -43,7 +43,7 @@ See the page on [Filebeat](/sending-data/data-shippers/beats/filebeat/) for furt

The above Filebeat configuration uses the [built-in parser `accesslog`](/sending-data/parsers/built_in_parsers/#accesslog).
The parser can parse logs formatted in the default Nginx log configuration.
If your log Nginx configuration is modified, create a [custom parser]({{< relref "parsing.md" >}}), by copying the accesslog parser and modifying it.
If your log Nginx configuration is modified, create a [custom parser]({{< relref "parsers/_index.md" >}}), by copying the accesslog parser and modifying it.
Then replace the parser name in the Filebeat configuration.

{{% notice note %}}
@@ -38,7 +38,7 @@ Since Docker just handles log lines from `stdout` as text blobs, you must parse
the lines to get the full value from them.

To do this, you can either use a built-in parser, or create new ones for your log
types. For more details on creating parsers, see the [parsing page]({{< ref "parsing.md" >}}).
types. For more details on creating parsers, see the [parsing page]({{< ref "parsers/_index.md" >}}).

{{% notice tip %}}
In terms of log management, Docker is just a transport layer.
@@ -4,15 +4,20 @@ category_title: Overview
weight: 100
---

Humio supplies built-in parsers for common log formats. For example it includes a parser for the widely-used [accesslog](https://httpd.apache.org/docs/2.4/logs.html#accesslog) format for web servers like Apache and Nginx.
Humio supplies built-in parsers for common log formats. For example it includes
a parser for the widely-used [accesslog](https://httpd.apache.org/docs/2.4/logs.html#accesslog)
format for web servers like Apache and Nginx.

This page lists and describes each of the built-in parsers.

When shipping data to Humio, check if there is a built-in parser for the logs before writing a custom parser.
When shipping data to Humio, check if there is a built-in parser for the logs
before writing a custom parser.
The built-in parsers are also a good starting point when creating custom parsers.

See the [Parsing Logs]({{< relref "parsing.md" >}}) page for an overview of how parsers work.
See the [parsing]({{< ref "parsers/_index.md" >}}) page for an overview of how parsers work.

You can examine each of the built-in parsers directly in the Humio UI. Just open its page and check the supported regular expression and timestamp formats. When you paste in test data Humio shows the result of parsing.
You can examine each of the built-in parsers directly in the Humio UI. Just
open its page and check the supported regular expression and timestamp formats.
When you paste in test data Humio shows the result of parsing.

{{%children%}}
@@ -8,7 +8,7 @@ It expects to find a JSON property called `@timestamp` containing a
[ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) formatted time string.

If you don't have control over the JSON format
you can [create a custom JSON parser]({{< relref "parsing.md#json-parser" >}}).
you can [create a custom JSON parser]({{< ref "json-parsers.md" >}}).

## Example Input
``` json
@@ -11,7 +11,7 @@ parser.

Go to the **Parsers** subpage in your repository to see all the available parsers:

{{% figure src="/images/parsersx.png" %}}
{{% figure src="/images/parsersx.png" width="600px" %}}

### Built-in parsers

@@ -24,7 +24,7 @@ You can also copy existing parsers to use as a starting point for creating new p

The following screenshots shows the **Parser** page with a custom parser called `humio`:

![Custom Parser`](/images/custom-parser.png)
{{% figure src="/images/custom-parser.png" width="600px" %}}

The **Parser** page lets you define and test parsers.

@@ -8,7 +8,7 @@ There are steps to getting your data into Humio:

1. [Generating an Ingest Token]({{< ref "ingest_tokens.md" >}}) Token - A special API token only for the Ingest API.
1. Sending data - Which is the subject of this page
1. Parsing and ingesting data - Described in the [Parsers sections]({{< ref "parsing.md" >}})
1. Parsing and ingesting data - Described in the [Parsers sections]({{< ref "parsers/_index.md" >}})

Sending data to Humio (also called _data shipping_) can be done in a couple of ways:

@@ -96,11 +96,11 @@ _Tip: If you are already using ElasticSearch ELK you can also take a look at how
While Humio has build-in support for the most popular logging formats (e.g. AccessLog, JSON)
and can rip out almost anything with the `kv` <!-- TODO: Missing Link --> parser, you still
may have special needs for your custom application logs. If that is the case you need to
[create your own custom parser]({{< relref "parsing.md" >}}).
[create your own custom parser]({{< relref "parsers/_index.md" >}}).

## 5. Next Steps

- [Query Function Reference]({{< relref "query-functions/_index.md" >}})
- [Creating Custom Parsers]({{< relref "parsing.md" >}})
- [Creating Custom Parsers]({{< relref "parsers/_index.md" >}})
- [Virtual Repositories]({{< relref "views.md" >}})
- [Controlling Retention]({{< relref "retention.md" >}})
@@ -87,7 +87,11 @@ the search documentation.

### Field: @timestamp {#timestamp}

The timestamp of an event is represented in the `@timestamp` attribute.
The timestamp of an event is represented in the `@timestamp` field. This field
defines where the event is stored in the Humio's database, and is what defined
wether an event is included in search results when searching a time range.

It needs [special treatment when parsing incoming data]({{< ref "creating-a-parser.md#parsing-timestamps">}}) at ingest time.

### Field: #repo {#repo}

@@ -96,4 +100,4 @@ This is useful in cross-repository searches when using [views]({{< ref "views.md

### Field #type {#type}

The type field is the name of the [parser]({{< ref "parsing.md" >}}) used to ingest the data.
The type field is the name of the [parser]({{< ref "parsers/_index.md" >}}) used to ingest the data.
@@ -12,7 +12,7 @@ Humio saves data in Data Sources. You can provide a set of Tags to specify which
You can add Tags to [Events]({{< ref "events.md" >}}) that you ingest into Humio.
Tags provide an important way to speed up searching. They allow Humio to select which Data Sources to search through.
For example, you can add Tags to Events that represent host names, file names, service names, or the kind of service.
Tags can be configured in [parsers]({{< ref "parsing.md" >}}) or specified in the APIs for data ingestion.
Tags can be configured in [parsers]({{< ref "parsers/_index.md" >}}) or specified in the APIs for data ingestion.

In Humio tags always start with a #. When turning a field into a tag it will be
prepended with `#`.
@@ -18,15 +18,15 @@ can find the integration you need. For example, if what you want is:

* **Logs from a Docker container**, then:
1. Start [here](integrations/platforms/docker.md), then
2. Get information about [how Humio parses logs]({{< relref "parsing.md" >}}).
2. Get information about [how Humio parses logs]({{< relref "parsers/_index.md" >}}).

* **Logs that an application writes to a file**, then:
1. Read an overview of the [Filebeat]({{< ref "filebeat.md" >}}) log shipper, then
2. Get information about parsing [here]({{< ref "parsing.md" >}})
2. Get information about parsing [here]({{< ref "parsers/_index.md" >}})

* **Metrics from platforms or applications**, then:
1. Read the [Metricbeat]({{< ref metricbeat.md >}}) topic, then
2. Get information about parsing [here]({{< ref "parsing.md" >}})
2. Get information about parsing [here]({{< ref "parsers/_index.md" >}})


## Start using Humio

0 comments on commit 05f15d0

Please sign in to comment.