Permalink
Browse files

update many of the logshippers to use the new ingest endpoints

  • Loading branch information...
chvitved committed Dec 4, 2018
1 parent c42be19 commit e2fcb2d95e9cd208d4f86ba6aa11af784c089307
@@ -54,7 +54,7 @@ You can use the following `elasticsearch` output configuration template:
``` yaml
output:
elasticsearch:
hosts: ["https://$HOST:443/api/v1/dataspaces/$REPOSITORY_NAME/ingest/elasticsearch"]
hosts: ["https://$BASEURL/api/v1/ingest/elastic-bulk"]
username: $INGEST_TOKEN
```

@@ -20,7 +20,7 @@ For the full documentation on FluentD please see the [official documentation](ht
Some of the most commons parameters in the [Elasticsearch Output Plugin](https://docs.fluentd.org/v1.0/articles/out_elasticsearch) are

* `host`: The hostname of your Humio instance.
* `port`: The port of where Humio is exposing the Elastic Endpoint. Don't forget to enable `ELASTIC_PORT` the [Configuration parameter]({{ ref "/configuration" }}).
* `port`: The port of where Humio is exposing the Elastic Endpoint. Don't forget to enable `ELASTIC_PORT` the [Configuration parameter]({{< ref "configuration" >}}).
* `scheme`, `ssl_version`: Depending on whether TLS is enabled on `host`:`port`, this should be set to either `https` or `http`. Humio Cloud has TLS enabled. In [some cases](https://github.com/uken/fluent-plugin-elasticsearch/issues/439) it is necessary to specify the SSL version.
* `user` and `password`: while `password` can be ignore, but must be present, `user` should be set to an [ingest token]({{< ref "/sending-data-to-humio/ingest-tokens.md" >}}).

@@ -95,7 +95,7 @@ The JSON with connector properties could look like below:
"schema.ignore": true,
"behavior.on.malformed.documents": "warn",
"drop.invalid.message": true,
"connection.url": "http://$HOST:$PORT/api/v1/dataspaces/$REPOSITORY_NAME/ingest/elasticsearch",
"connection.url": "http://$HOST:$PORT/api/v1/ingest/elastic-bulk",
"type.name": "kafka-ingest",
"max.retries": 1000
}
@@ -54,43 +54,24 @@ input{
}
output{
elasticsearch{
hosts => ["https://$BASEURL/api/v1/dataspaces/$REPOSITORY_NAME/ingest/elasticsearch/"]
hosts => ["https://$BASEURL/api/v1/ingest/elastic-bulk"]
user => "$INGEST_TOKEN"
password => "notused" # a password has to be set, but Humio does not use it
}
}
```

An important next step is [assigning a specific parser to the Ingest API Token]({{< ref "assigning-parsers-to-ingest-tokens.md" >}}).

For more information on parsers, see [parsing]({{< relref "parsers/_index.md" >}}).

### Adding tags to events

Please read [the section on tags]({{< ref "tagging.md" >}}) before adding tags
to your events. Add tags by including them in the "inputs/exec" section:

```
input{
exec{
command => "date"
interval => "5"
add_field => { "[@tags][customer]" => "CustomerName" }
}
}
```



{{< partial "common-rest-params" >}}


{{% notice warning %}}
Logstash uses 9200 as the default port if no port is specified. So if Humio is listening on the default ports 80 or 443 these ports should be explicitly put in the $BASEURL
{{% /notice %}}

In the above example, Logstash calls the Linux `date` command every
five seconds. It passes the output from this command to Humio.


### Field mappings

When you use the ElasticSearch output, Logstash outputs JSON
@@ -108,16 +89,21 @@ configuration looks like this:
Humio maps each JSON object into an Event. Each field in the JSON
object becomes a field in the Humio Event.

Humio treats some fields as special cases:
`@timestamp` is a special field in the Elastic protocol. It must be present, and contain the timestamp in ISO 8601 format (`yyyy-MM-dd'THH:mm:ss.SSSZ`).
It is possible to specify the timezone (like +00:02) in the timestamp. Specify the time zone if you want Humio to save this information.
Logstash adds the `@timestamp` field automatically. <br /><br />Depending on the configuration the timestamp can be the time at which Logstash handles the event, or the actual timestamp in the data.
If the timestamp is present in the data you can configure logstash to parse it, for example, by using the date filter.
Another option is to handle parsing the timestamp in Humio by connecting a parser to the ingest token.

### Adding Parsers in Humio
Humio can do further parsing/transformation of the data it receives by [connecting a parser to the ingest token]({{< ref "assigning-parsers-to-ingest-tokens.md" >}}).
For more information on parsers, see [parsing]({{< relref "parsers/_index.md" >}}).

| Name | Description |
---------------------------|---------------|
| `@timestamp` | This field must be present, and contain the timestamp in ISO 8601 format. This format is: `yyyy-MM-dd'THH:mm:ss.SSSZ`. <br /><br />You can specify the timezone (like +00:02) in the timestamp. Specify the time zone if you want Humio to save this information. Logstash adds the `@timestamp` field automatically. <br /><br />Depending on the configuration the timestamp can be the time at which Logstash handles the event, or the actual timestamp in the data. If the timestamp is present in the data you can configure logstash to parse it, for example, by using the date filter. |
| `message` | If present, Humio treats this field as the rawstring of the event. <br /><br />Humio maps this field to the `@rawstring` field which is the textual representation of the raw event in Humio. <br /><br />If you do not provide the message or rawstring field the rawstring representation is the JSON structure as text. |
| `rawstring` | This field is similar to the `message` field. <br /><br />If you provide both fields Humio uses the `message` field. The reason for having both is that some Logstash integrations automatically set a message field representing the raw string. <br /><br />In Humio, we use the name rawstring. |

### Dropping fields

Logstash often adds fields like `host` and `@version` to events. You
can remove these fields using a filter and the `drop_field` function
in Logstash.


@@ -1,5 +1,5 @@
---
title: "Others Data Shippers"
title: "Other Data Shippers"
weight: 1000
---

@@ -35,7 +35,7 @@ filebeat.inputs:
"@type": accesslog
output.elasticsearch:
hosts: ["https://$HOST:443/api/v1/dataspaces/$REPOSITORY_NAME/ingest/elasticsearch"]
hosts: ["https://$BASEURL/api/v1/ingest/elastic-bulk"]
username: $INGEST_TOKEN
```

@@ -46,7 +46,7 @@ See the page on [Filebeat](/sending-data/data-shippers/beats/filebeat/) for furt
The above Filebeat configuration uses the [built-in parser `accesslog`](/sending-data/parsers/built_in_parsers/#accesslog).
The parser can parse logs formatted in the default Nginx log configuration.
If your log Nginx configuration is modified, create a [custom parser]({{< relref "parsers/_index.md" >}}), by copying the accesslog parser and modifying it.
Then replace the parser name in the Filebeat configuration.
Then [connect the parser to the ingest token]({{< ref "assigning-parsers-to-ingest-tokens.md" >}}) or put its name as the value of the @type field in the Filebeat configuration.

{{% notice note %}}
***Response time***
@@ -148,7 +148,7 @@ metricbeat.modules:
processes: ['.*nginx.*']
output.elasticsearch:
hosts: ["https://$HOST:443/api/v1/dataspaces/$REPOSITORY_NAME/ingest/elasticsearch"]
hosts: ["https://$BASEURL/api/v1/ingest/elastic-bulk"]
username: $INGEST_TOKEN
```

@@ -68,14 +68,10 @@ metricbeat.modules:
period: 10s
output.elasticsearch:
hosts: ["https://$HOST:443/api/v1/dataspaces/$REPOSITORY_NAME/ingest/elasticsearch"]
hosts: ["https://$BASEURL/api/v1/ingest/elastic-bulk"]
username: $INGEST_TOKEN
```

Where:

* `$HOST` - is the name of your Humio server.
* `$REPOSITORY_NAME` - is the name of your repository on your server.
* `$INGEST_TOKEN` - is the [ingest token]({{< relref "ingest-tokens.md" >}}) for your repository.
{{< partial "common-rest-params.html" >}}

See also the page on [Beats]({{< relref "sending_data/data-shippers/beats/_index.md" >}}) for more information.

0 comments on commit e2fcb2d

Please sign in to comment.