Skip to content

Commit

Permalink
update readme [ci skip]
Browse files Browse the repository at this point in the history
  • Loading branch information
@HatsuneMiku3939
HatsuneMiku3939 committed Aug 25, 2018
1 parent 4296c5a commit 99bc68c
Showing 1 changed file with 222 additions and 75 deletions.
297 changes: 222 additions & 75 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,37 @@

[Fluentd](http://fluentd.org/) input plugin to pull log from rest api.

Many of modern server application offer status reporting API via http (even 'fluentd' too). This plugin will help to gathering status log from these status api.
Many of modern server application offer status reporting API via http (even 'fluentd' too).
This plugin will help to gathering status log from these status api.

* [Installation](#installation)
* [Usage](#usage)
* [Basic Usage](#basic-usage)
* [Monitor Status Code](#monitoring-http-status-code-only)
* [Override User Agent](#override-user-agent)
* [HTTP Basic Auth](#http-basic-auth)
* [HTTP Proxy](#http-proxy)
* [Logging Response Header](#logging-http-response-header)
* [Custom Request Header](#custom-request-header)
* [HTTPS Support](#https-support)
* [Configuration](#configuration)
* [tag](#tag-string-required)
* [url](#url-string-required)
* [agent](#agent-string-optional-default-fluent-plugin-http-pull)
* [interval](#interval-time-required)
* [format](#format-required)
* [status_only](#status_only-bool-optional-default-false)
* [http_method](#http_method-enum-optional-default-get)
* [timeout](#timeout-time-optional-default-10s)
* [proxy](#proxy-string-optional-default-nil)
* [user](#user-string-optional-default-nil)
* [password](#password-string-optional-default-nil)
* [response_header](#response_header-section-optional-default-nil)
* [request_header](#response_header-section-optional-default-nil)
* [verify_ssl](#verify_ssl-bool-optional-default-true)
* [ca_path](#ca_path-string-optional-defualt-nil)
* [ca_file](#ca_file-string-optional-defualt-nil)
* [Copyright](#copyright)

## Installation

Expand All @@ -32,87 +62,227 @@ And then execute:
$ bundle
```

## Example
## Usage

You can found more examples in `test/plugin/test_in_http_pull.rb`
### basic usage

In your Fluentd configuration, use @type http_pull.
`tag`, `url`, `interval`, `format` is mandatory configuration.

```
<source>
@type http_pull
tag status
url http://your-infrastructure/api/status.json
interval 1s
format json
</source>
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
```

In this example, a response of your infrastructure parsed as json.
A result contains `url`, `status`, `message`. `message` is a response of your infrastructure.

```
{
"url": "http://your-infrastructure/api/status.json",
"status": 200,
"message": {
// response of your infra structure
}
}
```

You can found more examples in this document.

### Monitoring http status code only

If you need only http status code, not response body, You can turn off response
body parser to set `status_only` is true. Remember that `format` is mandatory.
In this case, you must set `format` is none.


```
<source>
@type http_pull
@type http_pull
tag test
url http://www.google.com
interval 1s
tag fluentd.status
url http://your-infrastructure/healthcheck
interval 1s
format none
status_only true
status_only true
format none
</source>
<match test>
@type stdout
</match>
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/healthcheck","status":200}
```

### Override User Agent

You can set the User Agent by specifying the `agent` option.

# 2017-05-17 21:40:40.413219000 +0900 test: {"url":"http://www.google.com","status":200}
# 2017-05-17 21:40:41.298215000 +0900 test: {"url":"http://www.google.com","status":200}
# 2017-05-17 21:40:42.310993000 +0900 test: {"url":"http://www.google.com","status":200}
# 2017-05-17 21:40:43.305947000 +0900 test: {"url":"http://www.google.com","status":200}
```
<source>
@type http_pull
tag status
url http://your-infrastructure/api/status.json
interval 1s
format json
agent infrastructure_monitor
</source>
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
```

### HTTP basic auth

If your infrastructure is protected by HTTP basic auth,
You can use `user`, `password` options to provide authentication information.

### Monitoring fluentd itself
```
<source>
@type monitor_agent
@type http_pull
tag status
url http://your-infrastructure/api/status.json
interval 1s
port 24220
format json
user foo
passwrd bar
</source>
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
```

### HTTP proxy

You can send your requests via proxy server.

```
<source>
@type http_pull
@type http_pull
tag fluentd.status
url http://localhost:24220/api/plugins.json
interval 1s
tag status
url http://your-infrastructure/api/status.json
proxy http://your-proxy-server:3128
interval 1s
format json
format json
</source>
<match fluentd.status>
@type stdout
</match>
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
```

### Logging HTTP response header

If you wish to monitoring not only response body but also response header,
provide name of header in `response_header` sections.

# 2017-05-17 21:41:47.872951000 +0900 fluentd.status: {"url":"http://localhost:24220/api/plugins.json","status":200,"message":{"plugins":[{"plugin_id":"object:1e7e3d...
# 2017-05-17 21:41:48.955316000 +0900 fluentd.status: {"url":"http://localhost:24220/api/plugins.json","status":200,"message":{"plugins":[{"plugin_id":"object:1e7e3d...
# 2017-05-17 21:41:50.033628000 +0900 fluentd.status: {"url":"http://localhost:24220/api/plugins.json","status":200,"message":{"plugins":[{"plugin_id":"object:1e7e3d...
# 2017-05-17 21:41:51.107372000 +0900 fluentd.status: {"url":"http://localhost:24220/api/plugins.json","status":200,"message":{"plugins":[{"plugin_id":"object:1e7e3d...
```
<source>
@type http_pull
tag status
url http://your-infrastructure/api/status.json
interval 1s
format json
<response_header>
header Content-Type
</response_header>
<response_header>
header Content-Length
</response_header>
</source>
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
```


### Custom request header

The custom request header also supported.
This will help you when your infrastructure needs authentication headers or something like that.

### Monitoring elasticsearch cluster health
```
<source>
@type http_pull
@type http_pull
tag status
url http://your-infrastructure/api/status.json
interval 1s
tag es.cluster.health
url http://localhost:9200/_cluster/health
interval 1s
format json
format json
<request_header>
header API_ACCESS_KEY
value hatsune
</request_header>
<response_header>
header API_ACCESS_KEY_SECRET
value miku
</response_header>
</source>
<match es.cluster.health>
@type stdout
</match>
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
```


### HTTPS support

If your infrastructure has https endpoints, just use https url for monitoring them.

#### self-signed SSL


If your infrastructure has https endpoints secured by self signed certification,
you can provide custom certification file via `ca_path`, `ca_file` option.

# 2017-05-17 12:49:09.886298008 +0000 es.cluster.health: {"url":"http://localhost:9200/_cluster/health","status":200,"message":{"cluster_name":"elasticsearch","status":"green",...
# 2017-05-17 12:49:10.669431296 +0000 es.cluster.health: {"url":"http://localhost:9200/_cluster/health","status":200,"message":{"cluster_name":"elasticsearch","status":"green",...
# 2017-05-17 12:49:11.668789668 +0000 es.cluster.health: {"url":"http://localhost:9200/_cluster/health","status":200,"message":{"cluster_name":"elasticsearch","status":"green",...
# 2017-05-17 12:49:12.668789849 +0000 es.cluster.health: {"url":"http://localhost:9200/_cluster/health","status":200,"message":{"cluster_name":"elasticsearch","status":"green",...
```
<source>
@type http_pull
tag status
url https://your-infrastructure/api/status.json
interval 1s
ca_path /somewhere/ca/stored
ca_file /somewhere/ca/stored/server.crt
format json
</source>
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
```

And, disable SSL verification also supported. (not recommended)

```
<source>
@type http_pull
tag status
url https://your-infrastructure/api/status.json
interval 1s
verify_ssl false
format json
</source>
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
```

## Configuration

### Basic options
### Basic Configuration

#### tag (string) (required)

Expand Down Expand Up @@ -143,7 +313,7 @@ for more detail.

If `status_only` is true, body is not parsed.

###E http_method (enum) (optional, default: :get)
#### http_method (enum) (optional, default: :get)

The http request method for each requests. Avaliable options are listed below.

Expand All @@ -157,13 +327,13 @@ If `status_only` is true, `http_method` was override to `head`

The timeout of each request.

### Proxy options`
### Proxy Configuration

#### proxy (string) (optional, default: nil)

The HTTP proxy URL to use for each requests

### Basic auth options
### Basic Auth Configuration

#### user (string) (optional, default: nil)

Expand All @@ -173,7 +343,7 @@ The user for basic auth

The password for basic auth

### Req/Resp header options
### Req/Resp Header Configuration

#### response_header (section) (optional, default: nil)

Expand All @@ -183,7 +353,7 @@ The name of response header for capture.

The name, value pair of custom reuqest header.

### SSL options
### SSL Configuration

#### verify_ssl (bool) (optional, default: true)

Expand All @@ -195,33 +365,10 @@ The absolute path of directory where ca_file stored. Should be used with `ca_fil

#### ca_file (string) (optional, defualt: nil)

The Absolute path of ca_file. Should be used with `ca_path`.


## In case of remote error

### Can receive response from remote

```
{
"url": url of remote
"status": status code of response
"error": "RestClient::NotFound: 404 Not Found" or something similar
}
```

### All the other case

```
{
"url": url of remote
"status": 0
"error": "Errno::ECONNREFUSED: Connection refused - connect(2) for "localhost" port 12345" or something similar
}
```
The absolute path of ca_file. Should be used with `ca_path`.

## Copyright

* Copyright(c) 2017- filepang
* License
* Apache License, Version 2.0
* Apache License, Version 2.0

0 comments on commit 99bc68c

Please sign in to comment.