Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding HAProxy module #2384

Merged
merged 23 commits into from Sep 8, 2016
Merged
Changes from 11 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8c236c4
Merge pull request #1 from elastic/master
hartfordfive Jun 15, 2016
5b4adac
Merge pull request #2 from elastic/master
hartfordfive Jul 27, 2016
056f7f8
Committing current work for haproxy module
hartfordfive Aug 11, 2016
7b4c97d
Committing current changes.
hartfordfive Aug 13, 2016
8d9d696
Added haproxy package to fetch stat/info commands
hartfordfive Aug 16, 2016
dcf16a8
Updated stat/info metrics
hartfordfive Aug 16, 2016
16dafbe
Comitting current changes
hartfordfive Aug 21, 2016
ac6dbe1
Committing current changes
hartfordfive Aug 21, 2016
543319e
Updated necessary test related files
hartfordfive Aug 21, 2016
52652bf
Committing current changes
hartfordfive Aug 24, 2016
eb7c22d
Committing current changes
hartfordfive Aug 24, 2016
6c0eb6f
Fixed errors & small changes
hartfordfive Aug 27, 2016
e2828a8
Minor fixes
hartfordfive Aug 27, 2016
3e4384e
Fixed field names/values & removed unused function
hartfordfive Aug 29, 2016
004d9fa
Updated filebeat configs
hartfordfive Aug 29, 2016
007d4a4
Comitting curent changes
hartfordfive Aug 31, 2016
ef9c877
Merging in master & fixing conflicts
hartfordfive Sep 1, 2016
498ada4
Applied changes as per comments
hartfordfive Sep 2, 2016
e989718
Updated structure to follow event convetions
hartfordfive Sep 3, 2016
41eb385
Updated index templates
hartfordfive Sep 4, 2016
607c564
Fixed index templates and data type
hartfordfive Sep 4, 2016
6ba4fbd
Removed unecessary Sprintf & added experimental warning
hartfordfive Sep 5, 2016
e768cd2
Restored original version of file
hartfordfive Sep 7, 2016
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -7,6 +7,7 @@ beat:
- nginx
- redis
- zookeeper
- haproxy
environment:
- APACHE_HOST=apache
- APACHE_PORT=80
@@ -21,6 +22,8 @@ beat:
- MYSQL_PORT=3306
- ZOOKEEPER_HOST=zookeeper
- ZOOKEEPER_PORT=2181
- HAPROXY_HOST=haproxy
- HAPROXY_PORT=14567
- TEST_ENVIRONMENT=false
working_dir: /go/src/github.com/elastic/beats/metricbeat
volumes:
@@ -48,6 +51,9 @@ mysql:
nginx:
build: ${PWD}/module/nginx/_meta

haproxy:
build: ${PWD}/module/haproxy/_meta

redis:
image: redis:3.2.0

@@ -27,4 +27,5 @@ waitFor ${MYSQL_HOST} ${MYSQL_PORT} MySQL
waitFor ${NGINX_HOST} ${NGINX_PORT} Nginx
waitFor ${REDIS_HOST} ${REDIS_PORT} Redis
waitFor ${ZOOKEEPER_HOST} ${ZOOKEEPER_PORT} Zookeeper
waitFor ${HAPROXY_HOST} ${HAPROXY_PORT} HAProxy
exec "$@"
@@ -15,6 +15,7 @@ grouped in the following categories:
* <<exported-fields-apache>>
* <<exported-fields-beat>>
* <<exported-fields-common>>
* <<exported-fields-haproxy>>
* <<exported-fields-mongodb>>
* <<exported-fields-mysql>>
* <<exported-fields-nginx>>
@@ -443,6 +444,34 @@ required: True
The document type. Always set to "metricsets".


[[exported-fields-haproxy]]
== haproxy Fields

haproxy Module



[float]
== haproxy Fields




[float]
== stat Fields

stat



[float]
=== haproxy.stat.example

type: keyword

Example field


[[exported-fields-mongodb]]
== MongoDB Fields

@@ -0,0 +1,37 @@
////
This file is generated! See scripts/docs_collector.py
////

[[metricbeat-module-haproxy]]
== haproxy Module

This is the haproxy Module.



[float]
=== Example Configuration

The haproxy module supports the standard configuration options that are described
in <<configuration-metricbeat>>. Here is an example configuration:

[source,yaml]
----
metricbeat.modules:
- module: haproxy
metricsets: ["stat"]
enabled: true
period: 1s
hosts: ["localhost"]
----

[float]
=== Metricsets

The following metricsets are available:

* <<metricbeat-metricset-haproxy-stat,stat>>

include::haproxy/stat.asciidoc[]

@@ -0,0 +1,19 @@
////
This file is generated! See scripts/docs_collector.py
////

[[metricbeat-metricset-haproxy-stat]]
include::../../../module/haproxy/stat/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the metricset, see the
<<exported-fields-haproxy,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../module/haproxy/stat/_meta/data.json[]
----
@@ -61,6 +61,14 @@ metricbeat.modules:
# Password of hosts. Empty by default
#password: test123

#------------------------------- haproxy Module ------------------------------
- module: haproxy
metricsets: ["stat"]
enabled: true
period: 1s
hosts: ["localhost"]


#------------------------------- MongoDB Module ------------------------------
#- module: mongodb
#metricsets: ["status"]
@@ -43,4 +43,12 @@ metricbeat.modules:
# if true, exports the CPU usage in ticks, together with the percentage values

This comment has been minimized.

Copy link
@ruflin

ruflin Sep 5, 2016

Collaborator

Not sure how this change neded up in this PR?

This comment has been minimized.

Copy link
@hartfordfive

hartfordfive Sep 5, 2016

Author Contributor

I believe I got these changes when merging in the latest master the other day into my branch.

This comment has been minimized.

Copy link
@ruflin

ruflin Sep 6, 2016

Collaborator

Does not seem to be in master: https://github.com/elastic/beats/blob/master/metricbeat/etc/beat.yml Could you run git checkout master metricbeat/etc/beat.yml ?

cpu_ticks: false

#------------------------------- haproxy Module ------------------------------
- module: haproxy
metricsets: ["stat"]
enabled: true
period: 1s
hosts: ["localhost"]



@@ -205,6 +205,24 @@
type: integer
description: >
Total.
- key: haproxy
title: "haproxy"
description: >
haproxy Module
fields:
- name: haproxy
type: group
description: >
fields:
- name: stat
type: group
description: >
stat
fields:
- name: example
type: keyword
description: >
Example field
- key: mongodb
title: "MongoDB"
description: >
@@ -10,6 +10,9 @@ import (
// This list is automatically generated by `make imports`
_ "github.com/elastic/beats/metricbeat/module/apache"
_ "github.com/elastic/beats/metricbeat/module/apache/status"
_ "github.com/elastic/beats/metricbeat/module/haproxy"
_ "github.com/elastic/beats/metricbeat/module/haproxy/info"
_ "github.com/elastic/beats/metricbeat/module/haproxy/stat"
_ "github.com/elastic/beats/metricbeat/module/mongodb"
_ "github.com/elastic/beats/metricbeat/module/mongodb/status"
_ "github.com/elastic/beats/metricbeat/module/mysql"
@@ -0,0 +1,10 @@
2016-08-10T19:06:01-04:00 INFO Metrics logging every 30s

This comment has been minimized.

Copy link
@ruflin

ruflin Aug 26, 2016

Collaborator

The logs directory shuld not be commited. it is under .gitignore in the master branch

2016-08-10T19:06:01-04:00 INFO Loading template enabled. Reading template file: /Users/alain.lefebvre/Documents/code/github_personal/beats/metricbeat/metricbeat.template.json

This comment has been minimized.

Copy link
@ruflin

ruflin Aug 29, 2016

Collaborator

This file must be removed manually as it is already commited.

2016-08-10T19:06:01-04:00 INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /Users/alain.lefebvre/Documents/code/github_personal/beats/metricbeat/metricbeat.template-es2x.json
2016-08-10T19:06:01-04:00 INFO Elasticsearch url: http://localhost:9200
2016-08-10T19:06:01-04:00 INFO Activated elasticsearch as output plugin.
2016-08-10T19:06:01-04:00 INFO Publisher name: Alain-L-MBP
2016-08-10T19:06:01-04:00 INFO Flush Interval set to: 1s
2016-08-10T19:06:01-04:00 INFO Max Bulk Size set to: 50
2016-08-10T19:06:01-04:00 INFO Register [ModuleFactory:[system], MetricSetFactory:[apache/status, mongodb/status, mysql/status, nginx/stubstatus, redis/info, redis/keyspace, system/core, system/cpu, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, zookeeper/mntr]]
2016-08-10T19:06:01-04:00 CRIT Exiting: 1 error: metricset 'haproxy/stat' is not registered, module not found
@@ -61,6 +61,14 @@ metricbeat.modules:
# Password of hosts. Empty by default
#password: test123

#------------------------------- haproxy Module ------------------------------
- module: haproxy
metricsets: ["stat"]
enabled: true
period: 1s
hosts: ["localhost"]


#------------------------------- MongoDB Module ------------------------------
#- module: mongodb
#metricsets: ["status"]
@@ -177,6 +177,19 @@
}
}
},
"haproxy": {
"properties": {
"stat": {
"properties": {
"example": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
}
}
}
}
},
"metricset": {
"properties": {
"host": {
@@ -171,6 +171,18 @@
}
}
},
"haproxy": {
"properties": {
"stat": {
"properties": {
"example": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
},
"metricset": {
"properties": {
"host": {
@@ -36,13 +36,24 @@ metricbeat.modules:

# Per process stats
- process
enabled: true
enabled: false

This comment has been minimized.

Copy link
@ruflin

ruflin Aug 29, 2016

Collaborator

This change should be reverted

This comment has been minimized.

Copy link
@ruflin

ruflin Sep 6, 2016

Collaborator

@hartfordfive Can you please also revert this change?

period: 10s
processes: ['.*']

# if true, exports the CPU usage in ticks, together with the percentage values
cpu_ticks: false

#------------------------------- haproxy Module ------------------------------
- module: haproxy
metricsets:
- "stat"
- "info"
enabled: true

This comment has been minimized.

Copy link
@ruflin

ruflin Aug 29, 2016

Collaborator

haproxy should be disabled by default in the short config

This comment has been minimized.

Copy link
@ruflin

ruflin Sep 6, 2016

Collaborator

@hartfordfive This also should be set to false by default.

period: 10s
#stats_addr: "unix:///tmp/haproxy-stats.sock"
stats_addr: "tcp://127.0.0.1:14567"




#================================ General =====================================
@@ -0,0 +1,2 @@
FROM haproxy:1.6
COPY haproxy.conf /usr/local/etc/haproxy/haproxy.cfg
@@ -0,0 +1,11 @@
#- module: haproxy

#metricsets:
#- "stat"
#- "info"

#enabled: true

#period: 10s

#stats_addr: "tcp://127.0.0.1:14567
@@ -0,0 +1,7 @@
- module: haproxy
metricsets:
- "stat"
- "info"
enabled: true
period: 10s
stats_addr: "tcp://127.0.0.1:14567"
@@ -0,0 +1,4 @@
== haproxy Module

This is the haproxy Module.

@@ -0,0 +1,9 @@
- key: haproxy
title: "haproxy"
description: >
haproxy Module
fields:
- name: haproxy
type: group
description: >
fields:
@@ -0,0 +1,39 @@

global
# maximum number of simultaneous active connections
maxconn 4000
#daemon
user root
group staff

# for restarts
pidfile /var/run/haproxy.pid
# Logging to syslog facility local0
log 127.0.0.1 local0
stats socket 0.0.0.0:14567

This comment has been minimized.

Copy link
@ruflin

ruflin Aug 29, 2016

Collaborator

Is this something that is enabled by default in most haproxy setups? If not, could be worth to mention this in the haproxy module docs.

spread-checks 5
#debug

defaults

log global
mode http
balance roundrobin
maxconn 25000
option httplog
option abortonclose
option httpclose
option forwardfor
retries 3
option redispatch

timeout client 30s
timeout connect 30s
timeout server 30s
option httpchk HEAD /haproxy?monitor HTTP/1.0
timeout check 5s

listen http-webservices

bind 0.0.0.0:8888
server log1 127.0.0.1:8889 check
@@ -0,0 +1,4 @@
/*
Package haproxy is a Metricbeat module that contains MetricSets.
*/
package haproxy
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.